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Abstract. We study the Howe dualities involving the reductive dual pairs 
(0(d), spo(2m\2nj) and (Sp(d), osp(2m\2nj) on the (super)symmetric tensor of 
<C d eg C m l". We obtain complete decompositions of this space with respect to 
their respective joint actions. We also use these dualities to derive a character 
formula for these irreducible representations of spo(2m\2n) and osp(2m\2n) 
that appear in these decompositions. 
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1. Introduction 

Howe duality [TSJIH] relates the representation theories of a pair of Lie groups/ 
algebras. It enables the study of representations of one Lie group/algebra via the 
representations of its dual partner, and hence it has become a fundamental tool 
where representation theory of classical Lie groups/algebras is indispensable. As 
simple and fundamental a concept it is therefore of no surprise that the Howe 
duality also applies to generalizations of finite-dimensional Lie groups/algebras. 
We point out here the Howe dualities of finite-dimensional Lie superalgebras in 
gl 0122 Haim EH] of infinite-dimensional Lie algebras in (321 CHI 112 M and of 
infinite-dimensional Lie superalgebras in jH] . In the above-mentioned articles, the 
main themes revolve around the construction of Howe dualities. In the present 
article we are also concerned about applications of the Howe dualities that we 
obtain. 

Consider a Lie superalgebra whose representation theory we wish to study. 
Suppose that on some natural space one has a Howe duality involving this Lie 
superalgebra with a classical Lie group or Lie algebra as its dual partner. As the 
representation theory of its classical counterpart is well-understood, one expects 
that this should enable one to study the representations of the Lie superalge- 
bra in question with the help of the representation theory of its classical dual 
partner. Of particular interest is a derivation of a character formula for this 
Lie superalgebra. It appears plausible that knowing the character of the total 
space and the characters of each of the irreducible representations of the classical 
group /algebra, one should in principle be able to obtain a character formula for 
the Lie superalgebra in question. As is well-known, character formulas for Lie 
superalgebras in general are rather difficult to obtain, and hence such a method 
could facilitate the computation of characters for certain representations of Lie 
superalgebras. One of the main purposes of this paper is to demonstrate for the 
orthosymplectic Lie superalgebra that such an approach to character formulas is 
indeed viable. The general idea is the following. 

Let Q m be a classical Lie algebra of rank m and let X be a fixed finite- 
dimensional classical Lie algebra. Suppose on some space $ m the pair (g m ,X) 
forms a dual pair in the sense of Howe. Suppose that this is the case for every 
m. That is, we have for each m a (multiplicity-free) decomposition with respect 
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to g m x X of the form 

A 

where Vg m and V£' denote irreducible representations of Q m and X, respectively. 
Here A is summed over a subset of irreducible representations of Q m . Since here 
the correspondence between irreducible representations of g m and X, given by 
A — ► A', is one-to-one, we will write V£ for ■ 

Now suppose that g m \ n is the Lie superalgebraic analogue of g m and we have 
an action of the dual pair Q m \ n x X on $ m \ n , which is the tensor product of $ m 
with a Grassmann superalgebra depending on n. Thus we have similarly 

A 

where V n A , denotes an irreducible representation of Q m \ n . 

Our claim is that if one knows the characters of V^ m for every m, then one, in 
principle, also knows the characters for V n A . 

Let us now discuss the content of the present article in more detail. Let X = O 
or X = Sp so that X(d) denotes either the orthogonal or the symplectic group 
acting on C d . We have an induced action on C d <S> C m , thus giving rise to an 
action on the symmetric tensor S(C d <S> C m ). Now by classical invariant theory 
(c.f. E]) the invariants of X(d) in the endomorphism ring of S(C d ® C m ) is 
generated by quadratic invariants, which may be identified with the Lie algebra 
sp(2m) in the case X = O and so(2m) in the case X = Sp. This implies that 
(0(d), sp(2m)) and (Sp(d), so(2m)) are Howe dual pairs on S(C d ® C m ) . 

Now let C m ' n be the complex superspace of dimension (m\n). The Lie group 
X(d) acts in a similar fashion on the (super) symmetric tensor 5 , (C d ®C m '™). Anal- 
ogously one derives the (0(d), spo(2m\2n))- and the (0(d), osp(2m|2n))-Howe 
duality on S(C d <g>C m| "). Although these dualities appear already in Howe's clas- 
sical paper [T%] . the complete decompositions of S(C d ® C m|n ) with respect to 
these joint actions are unknown to the best of our knowledge. In j^Z] a partial 
decomposition is obtained for X = O, with a complete answer given in the case 
of m = n = 1 only. 

Our first main task is to give the complete decompositions of S^C^ ® C m ' n ) 
with respect to these Howe dual pairs. This is achieved in the following way. By 
[T3] the decomposition of S(C d <&C m \ n ) with respect to X(d) and its dual partner 
is reduced to the decomposition of the subspace of harmonic polynomials H with 
respect to the dual pair (X(d), gl(m\n)). Our task is then reduced to the con- 
struction of all (X (d) , gl(m\n))-highest weight vectors in H. Our analysis of the 
(X(d), g/(m|n))-highest weight vectors in H relies heavily on the (gl(d) , gl(m\n))- 
Howe duality in S(C d ®C m ^ n ) in jlj (see also [HOI EH]) and the description of their 
joint highest weight vectors given in ||. Another important ingredient is the 
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construction of an explicit basis for each irreducible gl(d) x gZ(m|n)-component 
that appears in S(C d ® C m|n ). 

The idea to obtain a character formula for the irreducible representations of 
spo(2m\2n) or osp(2m\2n) is roughly as follows. In order to simplify notation we 
take X = Sp in what follows, but note that the same applies to X = O with 
minor modification. We first consider the classical duality, i.e. the case when 
n = 0. Thus we have an identity of characters of the form 

chs(c d ®c m ) = Yl chV sm ® chK A o(2m) . 

A 

Since now characters are polynomial functions on a Cartan subalgebra, we can 

Write X5p(d)( X ) = chV Sp(d) aIld Xso {2m )(y) = Cn Ko(2m)> where X aIld Y deI1 ° te 

the linear functions on the respective Cartan subalgebras. The left-hand side 
is the character of the algebra of polynomials in dm variables, which is a sym- 
metric function in x and y. Now taking the limit as m — > oo in an appropri- 
ate way one obtains a combinatorial identity involving infinitely many variables 
y = yx, y2, ■ ■ ■ , y m , ■ ■ ■ ■ Since the right-hand side is symmetric in y, we may apply 
to this identity the involution u of symmetric functions that sends the complete 
symmetric functions to the elementary symmetric functions (see |24j). The uj 
turns the left-hand side into the character of the tensor product of a polyno- 
mial algebra with a Grassmann algebra. Therefore, due to "linear independence" 
of the Xs P (d)> ft follows that (modulo some minor manipulation of the variables) 
the expression ^(linim^oo Xs (2m)(y)) ^ essentially the character of the irreducible 
representation of osp{2m\2n) paired with Vgpuy At this point we wish to point 
out our results imply that the characters of the representations of the Lie super- 
algebra osp(2m\2n) (respectively spo(2m\2n)) , for any m,n G Z + , that appear 
under the Howe duality are completely determined by the characters of the rep- 
resentations of the Kac- Moody algebra corresponding to the infinite affine matrix 
(respectively C^) (see [TH]) that appear under a similar Howe duality. 

The next problem is to describe the expression Xs (2m) (y) • ^ or ^^^ s we use 
the beautiful formula of Enright [HI U\ for unitarizable irreducible representations 
associated to a classical Hermitian symmetric pair. The reason for this is that in 
our case we may express such a character in terms of Schur functions which are 
carried by uo to the so-called hook Schur functions of Berele and Regev [T. This 
allows us to obtain a satisfactory description of the characters. 

We now come to the organization of the paper. In Section^lwe recall some basic 
facts on the orthogonal and symplectic groups and the orthosymplectic superalge- 
bra, where we also take the opportunity to set the notation to be used throughout 
the paper. In Section El we recall the (gl(d), #Z(m|n))-duality on S(C d ® C m ' n ) 
and construct an explicit basis for each irreducible component that appears in 
the decomposition of S^C^ ® C m ' n ). In Section HJ and Section we study the 
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(0(d), spo(2m|2n))-duality and the (Sp(d), osp(2m|2n))-duality and obtain the 
complete decompositions of S(C d ® C m ' n ) with respect to their respective joint 
actions. In Section El we derive a character formula for these representations of 
spo(2m\2n) and osp(2m\2n) that appear in the decomposition of S(C d (g) C m ' n ). 
Here we should mention that in the case of 0(d) x spo(2m\2n) with d even, we 
are only able to derive the formula for a sum of two irreducible representations 
in general. We also remark that in a character formula for the oscillator 
representations is given. This corresponds to our case 0(1). In order to obtain 
a better description of the character formulas we are required to study Enright's 
formula in more detail. This is done in the Section In Section |S] we study the 
character formulas in more detail. In Section El as another application of our 
Howe dualities, we give formulas for decomposing tensor products of these irre- 
ducible spo(2m\2n)- and osp(2m\2n) modules that appear in the decomposition 
of S(C d ®C m \ n ). 

Finally all vector spaces, algebras etc. are over the complex field C unless 
otherwise specified. By a partition we mean a non-increasing finite sequence of 
non-negative integers. A composition is a finite sequence of either all non-negative 
integers or all positive half-integers. Furthermore, by a generalized partition we 
will always mean a finite non-increasing sequence of either all integers or all half 
integers. By a generalized composition we will mean a finite sequence of either all 
integers or all half integers. 

2. Parameterization of irreducible representations 

In this section we give parameterizations of irreducible representations of the 
Lie groups and Lie superalgebras that we will be dealing with in this paper. For 
a more complete treatment of the material on Lie groups the reader is referred 
to 0. 

2.1. Irreducible representations of the general linear Lie superalgebra. 

Let C m ' n denote the complex (m|n)-dimensional superspace. The space of com- 
plex linear transformations on C m ' n has a natural structure as a Lie superalgebra, 
which we will denote by gl(m\n). Choose a homogeneous basis for C m ' n so that we 
may regard gl(m\n) as (m + n) x (m + n) matrices. Denote by E^ the elementary 
matrix with 1 in the i-th row and j-th column and elsewhere. Then f) = ^ CEa 
is a Cartan subalgebra, while B = J2i<j ^Eij is a Borel subalgebra containing f). 
Recall that finite-dimensional irreducible g/(m|n)-modules are parameterized by 
A G (}* with Aj — A i+ i G Z + , for i = 1, • • • , m — 1, m + 1, • • • , m + n — 1, where 
Aj = X(Eii). We will denote the corresponding finite-dimensional irreducible mod- 
ule by V^ n . Suppose that A is a partition (or a Young diagram) with A m+i < n. 
Then drawing the corresponding diagram A may be visualized as lying in the 
(m|n)-hook, i.e. from n + 1-st column on the columns of A all have lengths less 
than m + 1. We may interpret A as a highest weight of gl(m\n) by associating to 
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the diagram A the labels A = (Ai, • • • , A m ; < \[ — m >, • • • ,< \' n — m >), where 
A- is the length of the i-th column of the diagram A, and < r > stands for r, 
if r G N, and otherwise. If clear from the context that A is a Young diagram 
with A r 
weight A. 



-l < n, we will mean by V^ n the irreducible g/(m|n)-module of highest 



2.2. Irreducible representations of the orthogonal group. Let us denote 
by {e 1 , • • • , e d } the standard basis for C d . Consider the symmetric non-degenerate 
bilinear form determined by the d x d matrix 



/0 




1\ 

1 



1 

\i o 




0/ 



The complex orthogonal group 0(d) is the subgroup of the complex general linear 
group GL(d) preserving this form. The Lie algebra of 0(d) is so(d), which consists 
of those A G gl(d) with J d A l J d + A = 0, that is, A is skew-symmetric with respect 
to the diagonal running from the top right to the bottom left corner. 

Consider the case when d — 2k is even. We take as a Borel subalgebra b 
the subalgebra of so(d) contained in the subalgebra of upper triangular matrices. 
Furthermore we take as a Cartan subalgebra of b the subalgebra f) spanned by 
the elements E u = En — for i — 1, • • • , k. Now a finite-dimensional 

irreducible representation of so(d) is determined by its highest weight A G f)* 
subject to 

X(En — E i+ i t i + i) G Z + , 

A(£'fc-i i fc-i + Ekk) G Z + , 

for i = 1, • • • ,k — 1. Let A« = \(En) and identify A with the sequence of 
complex numbers (Ai, • • • , Ajt). An irreducible representation of so(2k) is finite- 
dimensional if and only if its highest weight A satisfies the conditions Ai > A 2 • • • > 
Afe with either Aj G Z or else Aj G \ + Z, i — 1, • • • , k and Xj > 0, j — 1, • • • , k — 1. 
Furthermore such a weight lifts to a representation of SO(d) if and only if Aj G Z + . 

Let V be a finite-dimensional irreducible 0(<i)-module. When regarded as an 
so(<i)-module we have the following possibilities: 

(i) V is a direct sum of two irreducible so(<i)-modules of highest weights 
(Ai, A 2 , • • • , X k ) and (Ai, A 2 , • • • , X k -i, -X k ), respectively, where A fc > 0. 

(ii) V is an irreducible so(<i)-module of highest weight (Ai, A 2 , • ■ ■ , A^-i, 0). 

Here Aj G Z + for all i. In the first case, that is when V is the direct sum 
of the two irreducible so(<i)-modules we denote V by V^^, where we let A = 
(Ai,A 2 , ••• ,A fe _i,A fc > 0). In the second case there are two possible choices 
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of V, which we denote by Vq,« and Vq,^ ® det, respectively. Recalling that 
0(d) is a semidirect product of SO(d) and Z 2 these two 0(<f)-modules as SO(d)- 
modules are isomorphic. However as 0(d)-modu\es they differ by the determinant 
representation so that we may distinguish these two modules as follows: consider 
the element r G 0(d) — SO(d) that switches the basis vector e k with e k+1 and 
leaves all other basis vectors of C d invariant. We declare Void) ^° ^ e ^ e 0(d)- 
module on which r transforms an S'0(c/)-highest weight vector trivially. Note 
that r transforms an SO(d) -highest weight vector in the 0(<f)-module Vo(d) ®det 
by-1. 

We may associate Young diagrams to these 0(d)-highest weights as follows 
(cf. [II]). For Ai > A2 ■ ■ ■ > Afc > we have an obvious Young diagram of length 
k. When X k = 0, we associate to the highest weight of Vo(d) ^ ne usua l Young 
diagram of length less than k. To the highest weight of Void) ® ^ we ass °ci a te 
the Young diagram obtained from the Young diagram of A by replacing its first 
column by a column of length d— \' v Here and further, for a partition A, we denote 
by A' its conjugate partition. We have thus associated to each finite-dimensional 
irreducible representation of 0(d) a Young diagram A with X[ + A' 2 < d. 

Next consider the case when d = 2k + 1 is odd. We take as a Borel subalgebra 
b the subalgebra of so(d) spanned by upper triangular matrices so that a Cartan 
subalgebra f) of b is again spanned by the elements En = En — Ed+x-^d+i-i, 
for i — 1, ■ ■ ■ , k. Now a finite-dimensional irreducible representation of so(d) is 
determined by its highest weight A 6 f)* subject to 

X(Eu — Ei + i^ + i) E Z + , 
A(-E'fcfc) G 2^ + ' 

for i = 1, • • • , k — 1. We set Aj = \(En) and identify A with the sequence of 
complex numbers (Ai, A2, • • • , \k)- It follows that a highest weight A of so(2k + 1) 
gives a finite-dimensional irreducible representation if and only Ai > A2 • • • > Aj, 
and Aj G Z + or else Aj G \ + Z + , for i = 1, • • • , k. 

Recall that when d is odd 0(d) is a direct product of SO(d) and Z 2 . Thus 
any finite-dimensional irreducible representation of 0(d), when regarded as an 
SO (d)-module, remains irreducible. Conversely an irreducible representation of 
SO(d) gives rise to two non- isomorphic 0(<i)-modules that differ from each other 
by the determinant representation det. We let VAd) s t an d for the irreducible 
0(d) -module corresponding to A = (Ai > A 2 > • • • > A& > 0) on which the 
element — / transforms trivially so that {Vq^, Vq,* ® det} with A ranging over 
all partitions as above is a complete set of finite-dimensional non-isomorphic 
irreducible 0(<i)-modules. 

Similarly as before we may associate Young diagrams to these 0(<i)-highest 
weights. For the highest weight A = (Ai > A 2 • • • > A& > 0) of Vq,* we have an 
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obvious Young diagram with Z(A) := X[ < k. To the highest weight of <S> 
det we associate the Young diagram obtained from the Young diagram of A by 
replacing its first column by a column of length d — X[. 

Let £j G ()* so that £i(Ejj) = 5^. We put Xj = e ei when dealing with characters 
of 0(d). 

2.3. Irreducible representations of the symplectic group. Let d = 2k and 

consider the non-degenerate skew-symmetric bilnear form < -|- > given by the 
d x d matrix 

( J k \ 
\-Jk o J ■ 

The symplectic group Sp(d) is the subgroup of GL(d) preserving < -|- >. We 
take as a Borel subalgebra b the subalgebra of sp(d) that is contained in the 
subalgebra of upper triangular matrices and a Cartan subalgebra of b as the 
subalgebra i) spanned by the elements En = Eu—Ea+i-^d+i-i, for % = 1, • • • , k. A 
finite-dimensional irreducible representation of sp(d) is determined by its highest 
weight A G f)* subject to 

X(Eu — Ei+ij+i) G Z + , 
X(E k>k ) G Z + , 

for i — 1, ■ ■ ■ , k — 1. As before we let A« = X(Ea) and identify A with the sequence 
(Ai,A2, •"' ,Afc). A highest weight A of sp(2k) gives a finite-dimensional irre- 
ducible representation if and only if Ai > A 2 ■ ■ ■ > A^ and Aj G Z + for i — 1, ■ ■ ■ , k. 
Furthermore each such representation lifts to a unique irreducible representation 
of Sp(d) and so we obtain an obvious parameterization of 5p(d)-highest weight 
in terms of Young diagrams A with Z(A) < |. 

We let 6j G ()* so that €i(Ejj) = Sy. We put = e ei when dealing with 
characters of Sp(2k). 

2.4. Irreducible representations of the ortho-symplectic Lie superalge- 
bra. Let C m|n be the (m|n)-dimensional complex superspace. Suppose that n is 
even and (-|-) is a supersymmetric non-degenerate bilinear form, i.e. it is sym- 
metric on the even subspace C m '° and symplectic on the odd subspace C°' n . The 
orthosymplectic Lie superalgebra osp{m\n) (cf. [To] ) is defined to be the subalge- 
bra of gl{m\n) = gl(m\ri)o © gl(m\n)i consisting of those linear transformations 
preserving the form (-|-), i.e. osp(m\n) = osp(m\n)Q © osp(m\n)i with 

osp(m\n) e = {Ae gl(m\n) e \(Av\w) + (-l) tAcsv (v\Aw) = 0}, 

where v and w are any homogeneous vectors of C m ' n , degf here and further 
denotes the degree of the homogeneous element v and e G Z 2 . We will fix the 
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bilinear form associated to matrix 



Jm 














Jn/2 





— Jn/2 






We note that osp(m|n)g = so{m) © spin). Let b be a Borel subalgebra of 
osp{m\n) containing the Borel subalgebras of so{m) and spin) as chosen above 
so that a Cartan subalgebra f) of osp(m\n) can be taken to be the subalgebra 
spanned by the diagonal matrices E u = En — E m+ x-i,m,+\-ii i — 1> • • ' > 
£'[m] +J - j [m] +J - = E m+ j >m+ j — £ m+n+ i-j, m+ „ + i-j, j = 1, • " )f- Here and further 
the symbol [r] stands for the largest integer smaller than or equal to r. As 
usual, highest weight irreducible representations of osp(m, n) are parameterized 
by A G ()* and we denote by A« the i-th label X(En), for i = 1, • • • , [y] + |. As 
usual, we will identify A with (Ai, A2, • • • ). 

Suppose that m is an even integer and consider the following Z-gradation of 
osp{m\n). Let C m| ° = V © V* be a sum of two isotropic subspaces of C m ' 
with respect to the restriction of the form (-|-) on C m '°. Likewise let C°' n = 
W © W* be such an isotropic decomposition of C°' ra . We have osp(m|n)g = 
,S 2 (C 0|n ) © A 2 (C m|0 ) and osp(m\n)i = C m l° © C°l n . Set g = (V © W) © (V © 
01 = ,S 2 (\/) © A 2 (W^) © (V © W) and _x = ,S 2 (\/*) © A 2 {W*) © (V* © W*). This 
equips osp(m|n) with a Z-gradation with g isomorphic to ^(y|f) such that its 
standard Cartan subalgebra is also f). 

Now take a finite-dimensional irreducible o - m odule Vin of highest weight 

2 1 2 

A G f)*, which we again will identify with a sequence (Ai, A2, • • • )• We may extend 

Vin trivially to a module over the parabolic subalgebra floffifli- Inducing it to an 

2 1 2 

osp(m I n)-module, it is clear that it has a unique irreducible quotient, which we 
will denote by / . v Of course , . is not finite-dimensional in general. As 

■J osp(m\n) osp(m\n) o 

such osp(m|n)-modules play an important role in the sequel, we will give a more 
detailed description of their parameterizations. Let q G f)*, % — 1, • • • , [y] + 1, be 
defined by €i(Ejj) = 5ij. We will label the simple roots and coroots of osp(m\n) 
according to the following diagram. 




Here Ot\ = — €\ — 62lQ ; 2 = e l — e 2 ? ' - ' , CKM =ein_l — e™,-- - , Ol m+n = 6 m+n -, — 6 m+n , 

2 2 2 2 2 2' 

and, as is customary, (^) denotes an isotropic root. Thus if A = (Ai, A 2 , • • • , Am+n) 

mln- 
2 I 2^ 



2 

is the highest weight of a finite-dimensional irreducible ||)-module Vin, 



2 1 2 
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then the labels of the irreducible highest weight module V^,.^ with respect to 
the above Dynkin diagram is given by 

(2.1) (— Ai — A 2 , Ai — A 2 , • ■ • , As._i — A™, A™ + A« + i, A«+i — A™ +2 , ■ ■ ■ )• 

When dealing with characters of osp(m\n) we will use the notation Xj = e ej , for 
j = 1, • • • , y and Zi = e^ + \ for I — 1, ■ ■ • , |. 

On the superspace C m l" with m even we may take a skew-supersymmetric 
non-degenerate bilinear form (-|-), i.e. it is symplectic on the even subspace 
C m '° and symmetric on the odd subspace C°' n . In the same fashion we may 
define the symplectic-orthogonal Lie superalgebra spo(m\n) to be the subalge- 
bra of gl(jn\n) preserving (-|-). We remark that as Lie superalgebras we have 
spo(m\n) = osp(n\m) and hence our discussion of the ortho-symplectic Lie su- 
peralgebra carries over to spo(m\n), for n even, with minor modification. We 
label the simple roots and coroots according to the following diagram. 

oo o^s^o o 

Oil a 2 a f «f + l Q; f+2 Olm±n 

Here ot\ = — 2ei, tt 2 = 6i — £ 2 , ■ ■ ■ , om± = em—i — e™ , ■ • • , ot m+n = e -m+n -, — e m+n . 

2 2 2 2 2 2 

Similarly we will denote the irreducible quotient of the induced g/(m|n)-module 
Van by V S p o(m|n) . So if A = (Ai, A 2 , ■ ■ • , Am±n) is the gl{f\^ )-labels of V||„, then 

the spo(m | n)-labels of ^ S p ( m | n ) are 

(2.2) (— Ai, Ai — A 2 , • • • , Am _i - Am, A™ + A<zt+i, A« + i - A^ +2 , ■ ■ • ). 

When dealing with characters of spo(m\n) we will use the notation yj = e €j , for 
j = 1, • • • , tt and = e e ^ +i , for / = 1, • • • , |. 

3. The (gl(d), gl( y m\n))-'DUALiTY 

In this section we present some results on (gl(d), gZ(m|n))-duality that will be 
used later on. In particular, Theorem 13.41 constructs explicit bases for irreducible 
gl(d) x gZ(m|n)-modules appearing in the decomposition of S(C d ® C m ' n ), and we 
believe the result to be new. 

Consider the natural actions of gl(d\q) on C d ' 9 and gl(m\n) on C m K We can 
form the gl(d\q) x g/(m|n)-module C d ' 9 (S>C m ' n . We have an induced action on the 
symmetric tensor S(C d ^ q (g) C m ' n ). This action is completely reducible and in fact 
(gl(d\q), gl{m\n)) is a dual pair in the sense of Howe jl] (see also jHO])- Since in 
this paper we will only concern ourselves with the case when q = 0, we will make 
this assumption in what follows. In this case we have the following decomposition 

(3.1) S(C d ® C™l») - v * ® Kiln. 

A 
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The sum in (J3.1J) is over all partitions of integers A = (Ai, A 2 , • • • , \d) of length 
/(A) not exceeding d subject to A m+i < n. Since /(A) < d we may regard A as 
a highest weight for an irreducible gZ(<i)-module so that there is no ambiguity in 
V^- The meaning of V\ as a gZ(m|n)-module was explained in Section ETT1 

In the sequel it is important to have an explicit formula for the joint highest 
weight vectors of the irreducible component V^®V^ n in ([3.1)1 . (See also [2H] and 
[23] for different descriptions of these vectors.) In order to present them we need 
to introduce some more notation. 

We let e 1 ,...,e d denote the standard basis for the standard g/(<f)-module. 
Similarly we let e±, . . . , e m ; fi, . . . , f n denote the standard homogeneous basis for 
the standard gZ(m|n)-module. The weights of e\ t\ and fk are denoted by e\ 
and 6k, for 1 < i < d, 1 < I < m and 1 < k < n, respectively. We set 

(3.2) x\ := e* ® e h rf k := e* ® 

We will denote by C[x, 77] the polynomial superalgebra generated by ([3.2)1 . By 
identifying S , (C d ®C m ' n ) with the polynomial superalgebra C[x, 77] the commuting 
pair (gl(d), gl(m\n)) may be realized as first order differential operators as follows 
(1 < i, i' < d, 1 < s, s' < m and 1 < k, k' < n): 

m n n a 

(3-3) E%+E%> 

j=l J j=l 'J 

d n d r\ d r\ d r\ 

(3.4) yvJL V^ s — , Y^vi— ■ 

Here ([3.3)1 spans a copy of gl(d), while ([3.4)1 spans a copy of gZ(m|n). 

The standard Cartan subalgebras of gl(d) and gl(m\n) are spanned, respec- 
tively by 

m n d ^ d ^ 

E4^-+Efe and E- r ^- E'/a^j- 

j=i ax j j=i j=i 0,3:5 j=i °% 

while the nilpotent radicals are respectively generated by the simple root vectors 



m d n d 

i= i 3 j=1 or ij 



and 



E — 7' E 'Tfc-i — 7' E x ™ — 7' l<s<m, 1 < k <n. 
~ dxi drf k pi dr]{ 

We will consider two separate cases, namely m> d and m < d. 
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First suppose that m > d. Here the condition A m+i 
1 < r < min(d, m) define 



< n is vacuous. For 







(x\ 


x\ 




(3.5) 


A r := det 












w 


7* 

x 2 





Theorem 3.1. [1] In the case when m > d, the gl(d) x gl{m\n) highest weight vec- 
tors in C[x, rf\ associated to the weight A is given by the product Ay 2 • • • A^ . 

We now consider the case d > m. It is readily checked that the highest weight 
vectors associated to Young diagrams A with A m+ i = can be obtained just as 
in the previous case so that we may assume that /(A) > m. Let A' 1; X' 2 , . . . , X' x 
denote its column lengths as usual. We have d > \[ > \' 2 ■ ■ ■ > A^ and m > \' n+1 . 
For m < r < d, consider the following determinant of an r x r matrix: 



(3.6) 



*fc,r • 



det 



x 2 


xj ■ 

x 2 


T 

x 2 




/v>2 




vl 
vl 


vl ■ 
vl ■ 


V 

■ Vk 

■ vl 


[vl 


vl ■ 


■ vl) 



n. 



for j 



1, . . . , m, in 



That is, the first m rows are filled by the vectors (xj, . . . 

increasing order and the last r—m rows are filled with the same vector (jjI, . . . , r]Q. 
Here the determinant of a matrix 



A : = 


fa\ 

a] 


a\ ■ ■ 
a\ ■■ 


■ <A\ 




w 


a 2 r ■ ■ 





with matrix entries possibly involving Grassmann variables rf k , is by definition 



the expression Ylaes (~ l) 1 ^ a^ a 2 ^ ■ ■ ■ ar (r \ where 1(a) is the length of a in 
the symmetric group S r . 

Theorem 3.2. [4^ In the case when m < d, the gl(d) x gl(m\n) highest weight 
vectors in C[x, rj\ associated to the weight A is given by the product 

v Ai 

( 3 - 7 ) ri A ^ n a ^ 

k=l j=r+l 

where v is defined by \' u > m and \' u+1 < m. 
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For application purposes it is useful to construct an explicit basis for the gl(d) x 
g/(m|n)-modules V^ n that appear in the decomposition of S(C d g> C m|n ). This 
we will do now. 

Recall that A is a partition (or a Young diagram) which lies in the (m\n)- 
hook of length not exceeding d. Let xi, • ■ • ,x m and 771, •• • ,T] n be even and odd 
indeterminates, respectively. We form a tableau of shape A by filling the nodes of 



A from the set {x±, 



, Xr 



, 771 , • • - , r] n } so that the resulting tableau T is 



m\n) 



semi-standard. This means that we first fill the nodes of a sub-diagram fi C 
A with the even indeterminates {x±, • ■ ■ ,x m } so that the resulting sub-tableau 
is semi- standard. Then we fill the skew-diagram A//i with odd indeterminates 
{r]i, ■ ■ ■ ,rj n } so that its transpose is semi- standard. Let us suppose that the z-th 
column of T has length r and is filled from top to bottom by 



(3.8) (a^jj , Xi 2 , • • ■ , Xi s , T/j 1 , • 

We associate to ()3.8|) the following determinant 



(3.9) 



det 



1 2 



■nl 



is 



xni vl 



■Vjt 



x l\ 



x l 



where r = s + t. We set A T = Af . 

Theorem 3.3. The set {A T }, with T running over all (m\ri)- semi- standard 
tableaux of shape \, is a basis for the space of glid) -highest weight vectors in 
S(C d ® C m l n ) of highest weight A. 

Proof. It is easy to see that every A T is a gl(d)-highest weight vector of gl(d)- 
highest weight A. Now according to [I] the dimension of V^ n equals the number 
of (m|n)-semi-standard tableaux of shape A and hence it is enough to show that 
the set {A T } is a linearly independent set. Now due to weight considerations 
it is enough to prove that the set of {A T }, where T is over all (m|n)-semi- 



standard tableaux with fixed occurrence of {x±, 



r) n }, is linearly 



independent. We proceed by induction on the number of odd indeterminates 
that occur inside the T's. If that number is zero, then the conclusion of the 
theorem is know to be true (see e.g. [ID]). Thus we may assume that at least one 
odd indeterminate occurs in all of the T's. 
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Now let r)i be the odd indeterminate appearing in all A T with % minimal. Let 
(3.10) J]A T A T = 0. 

T 

We embed S(C d <g> C m| ") into S(C d <g> C m+1|n ) so that we may regard (l3~TTH) as a 
sum in S(C a! <g> C m+1|n ). We apply to (jSHDj) the linear map 

^ 

~t dm 

It is clear that the resulting sum is of the form 

E E 

where is the set of all tableaux obtained from X by replacing one of the rji- 
nodes by an x m+1 -node. We may assume that all S are (m + l|n)-semi-standard 
with one less odd node. Furthermore each As is a non-zero positive integral 
multiple of At- (Note that As = p\s if and only if rji appears with multiplicity 
p in some column and S is obtained from X by replacing the first rji node of this 
column by x m +\.) 

We claim that all S are distinct (m + l|n)-semi-standard tableaux and thus by 
induction they are linearly independent. This implies As = and hence At = 
and we are done. 

In order to prove the claim we consider two cases. 

In the first case suppose that S and S' are obtained from the same X. But in 
this case S and 5" are obviously different, since S and S' are obtained from X by 
replacing rji by x m+ i in different columns. 

Now suppose that S and 5" are obtained from T and T', respectively, and 
T ^ X". If the positions of i]i in T and T 1 are the same, then T and X" differ at 
some t] s node, i ^ s. But then S and 5" also differ at this particular ^-node as 
well. If on the other hand X and X' differ at some 77, node, then this means that X 
at a node has rji, while at the same node X' has some r] s , i 7^ s, or Xk, k 7^ m + 1. 
But then in all of S' this particular node is always r] s or x^, while in all S this 
particular node is either rji or x m+ \. Thus S and S' must be distinct. □ 

Let A be a Young diagram lying in the (m|n)-hook of with /(A) < d and X be 
an (m|n)-semi-standard tableau of shape A. We may fill the boxes of the Young 
diagram A from the set {x 1 , ■ ■ ■ , x d } in a way so that the resulting tableau X' is 
semi-standard. Let the z-th column of X' be filled by {x , • • • , x hr }. Suppose that 
a joint gl(m\n) x g/(<i)-highest weight vector is of the form (|3.7j) . We may replace 
the upper indices 1, 2, • • • , r of all the entries in A iiA ^ (or A\>_) by hi, k 2 , ■ ■ ■ ,k r . 
Let us call the resulting determinant Aj' x , (or Ay) and consider the following 
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product of determinant. 

LT^i I I a- 

k=l j=v+l 

It is clear from symmetry between the upper and lower indices that when T' 
ranges over all semi-standard tableaux we obtain a basis for the g/(m|n)-highest 
weight vectors of highest weight A in S(C d <S> C m ' n ). 

Now let A be a Young diagram lying in the (m|n)-hook of with /(A) < d and T 
and T" as before. Let the z-th column of T' and T be filled by 

{x k \---,X k r} 

(Xjj , Xi 2 , ■ • ■ , Xi 3 , Tjj 1 , • ■ ■ , Tjj t ) , 

respectively, from top to the bottom. To the i-th column of the bi-tableau (T, T') 
we associate the following determinant 







K 




■ x h\ 

X 12 






(3.11) 


Af ' r) := det 


x kl 
i 1 


I'S 

71 k2 ■ 

¥ 


■ x kr 

I'S 

■ i 






c 


rfa ■ 
132 








{<& 


71 k2 ■ 

ht 


■ tJ 



where again r = s+t. We set A( T,T ') = YiiLi ^-T' T ^ The following theorem gives 
an explicit basis for each irreducible gl(d) x g/(m|n)-component in S(C d ® C m ' n ). 

Theorem 3.4. The set A^ T ' T '\ where V is semi-standard in {x 1 , ■ ■ • , x d } and 
T is (m\n) -semi-standard in {x±, ■ ■ • , x m , 771, •• • , r/ n }, is a basis for ® V^ n in 

S(C d ®C m l n ). 

Proof. Given A^ T ' T " > with (T, T") fixed. By Theorem 13 . 31 and the Jacobson density 
theorem (more precisely by Burnside's theorem) we can find an element (a® b) G 
U(gl(d)) <g) U(gl(m\n)) such that (a®6)A^ T,T ^ is the joint gl(d) x #Z(m|n)-highest 
weight vector and a <S> b annihilates all A^ s ' s '\ for (S, S') 7^ (T, T"). This implies 
that the set {A^ T,T ^} is linearly independent. But the number of semi-standard 
tableaux in {a; 1 , - •• ,x d } times the number of (m|n)-semi-standard tableaux in 
{xi, ■ ■ ■ ,x m ,r]i,--- ,r) n } is precisely the dimension of the space V/ ® V\ n . □ 

Remark 3.1. The above theorem is known in the case when n = (see e.g. |Tu]). 
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4. The (O, spo)- and (Sp, osp)-duality 

Let C d be the d- dimensional complex vector space with standard basis {e 1 , e 2 , 
• ■ • , e d }. Let 0(d) be the orthogonal group leaving invariant the symmetric bilin- 
ear form (-|-) as in Section|21 and let C m ' n be the superspace of dimension (m\n). 
The natural action of 0(d) on C d extends to an action on C d ® C m ' n . This action 
gives rise to an action of 0(d) on the supersymmetric tensor S'(C d ®C m ' n ), which, 
as in Sectional we identify with C[x, 77], the commutative superalgebra in (|3.2|) . 
As the action gl(d) under this identification gets identified with certain first order 
differential operators as in ()3.3)1 . the action of the Lie algebra of 0(d) naturally 
gets identified with certain first order differential operators as well. 

Consider the following first order differential operators 

j=l udj i j=l u 'lk 

where i, s — 1, • • • , m and k, t — 1, - • • , n. It is evident that they form a basis for 
the Lie superalgebra gl(m\n) and it is clear that 0(d) commutes with gl(m\n). 
Next consider another set of operators on C[x, rf\. 

d d d 

Ojxx _ J~d+l-j Otx V _ST^ Jd+l-j OtVV J^d+l-j 

j=l j=l j=l 

where 1 < i < s < m and 1 < k < t < n. We note that these operators 
also commute with the action of 0(d) on C[x, rj\. It is not hard to see that 
these operators together with (j4.1|) form a basis of the symplectic-orthogonal Lie 
superalgebra spo(2m\2n). In fact, using the Z-gradation of spo(2m\2n) given in 
Section0 we have 0i = £ C°Aff + £ C°A^ + £ C° A% and _i = £ C°If s x + 
+ ^°Iki- Thus on C[x, rf\ we have an action of 0(d) x spo(2m\2n) . 
An element / G C[x, 77] will be called A-harmonic, if Af®f — A x J]f = 
°Ay / = 0. The space of °A-harmonics will be denoted by Note that since 
[gl(m\n), Qi] C 01 the space is invariant under the action of gl(m\n). Also 

H is clearly invariant under the action of 0(d). Hence we have an action of 
0(d) x gl(m\n) on °H. Let °I be the subalgebra of C[x, 77] generated by °If s x , 

Ift and I'm - It is clear that I is the subalgebra of 0(<i)-invariants in C[x, 77] . 
We have the following theorem. 
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Theorem 4.1. jT3j The pairs (0(d), spo(2m\2n)) and (0(d), gl(m\n)) form dual 
reductive Howe pairs on S(C d (g) C m ' n ) and on °H , respectively. Thus we have 



'[ X 5 V] — E ^O(d) ® Kpo(2m|2re)> 
A 

H = Y. V o^)®V^ n , 



where X is summed over a set of irreducible 0(d)-highest weights. Here X' and X" 
are certain non-isomorphic irreducible spo(2m\2n)- and gl(m\n) -highest weights, 
respectively. Furthermore the map I ® H — > C[x, rf\ given by multiplication is 
surjective and we have, for each X, V^' o(2m | 2n) = °IV^ n . 

Let d be an even integer and consider the <i-dimensional complex vector space 
with the standard basis e 1 , e 2 , • ■ ■ ,e d and equipped with the non-degenerate skew- 
symmetric bilinear form < •)• > as in Section Let Sp(d) be the corresponding 
symplectic group. Again we have an action of Sp(d) on C d <8> C m ' n , inducing 
an action of Sp(d) on the supersymmetric tensor S(C d (g) C m '™), which we again 
identify with C[x, rj\. 

Introduce the following operators 

d d 

2 2 

Spjxx _ \ ^ (i d+l-j _ d+l-j j\ Spjxr) _ \ ^ fjj _ d+l-j d+l-j\ 

is / j y^i^s ^sll 1 ik / j \ Jj i'lh 'Ik ji 



3=1 3=1 

d d 
2 . 2 



Spjrm _ \- ( J d+l-j _ d+l-j j\ Sp A xx _ \- ( g g g g \ 

5p A^ = E 



d d d d 



Sp *r t = E 



d d d d 



94 drjt +1 ~ j dri d+1 ~ 3 drf t 



where 1 < i < s < m and 1 < k < t < n. It is again not hard to see that these 
operators together with (j4.1|) form a basis for the Lie superalgebra osp(2m\2n) 
and their actions and that of Sp(d) on C[x, rj\ commute. 

An element / G C[x,r?] will be called Sp A-harmomc, if Sp A™f = Sp A^f = 
Sp/±vvf _ q fj^g S p ace f •S'PA-harmonics will be denoted by Sp H. Similarly 
we have an action of Sp(d) x gl(m\n) on Sp H. Let Sp I be the subalgebra of 
C[x, 77] generated by Sp If s x , Sv l^ and Sp I^ so that Sp I is the subalgebra of Sp(d)- 
invariants in C[x, rj]. In a similar fashion we have the following theorem. 
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Theorem 4.2. jTSj The pairs (Sp(d),osp(2m\2n)) and (Sp(d), gl(m\n)) form 
Howe dual reductive pairs on S(C d (g) C m ' n ) and on Sp H , respectively. Therefore 
we have a decomposition of modules 

C[x, 77] = ^2 Ksp(d) ® K>sp(2m|2n)> 
A 

A 

where X is summed over a set of irreducible Sp(d) -highest weights. Here X' and X" 
are certain non-isomorphic irreducible osp(2m\2n) - and gl(m\n) -highest weights, 
respectively. Furthermore the map Sp I® Sp H — > C[x, 77] given by multiplication is 
surjective and we have, for each X, V^ (2m | 2n) = Sp IV^ n . 

The proofs of Theorem 14.11 and Theorem 14.21 are based on the fact that the 
invariants of the classical group of the corresponding dual pair in the endomor- 
phism ring of S(C d ® C m ' n ) are generated by quadratic invariants. Although in 
[T3] it is shown that the pairs (0(d), spo(2m\2n)) and (Sp(d),osp(2m\2n)) are 
indeed dual pairs on S(C d <g> C m|n ), the explicit decomposition of S(C d <g> C m|n ) 
was not given. We will embark on this task in Sectional 

We conclude this section by showing that the representations of spo(2m\2n) 
and osp(2m\2n) that appear in Theorem 14.11 and Theorem 14.21 are unitarizable. 
We first recall some definitions. 

Let A be a superalgebra and > an anti-linear map with (aby = b^a^ , for a, b in A. 
We call ' an anti-linear anti-involution if (a')' = a. Now let A be a superalgebra 
equipped with an anti-linear anti-involution < and let V be an A-module. A 
Hermitian form (-|-) on V is said to be contravariant if (av\w) = (v\a^w), for 
a G A and v,w G V. If furthermore (-|-) is positive-definite, then V is said 
to be a unitarizable A-module. We remark here that we have defined the anti- 
involution and the contravariant form without "super signs" . It follows then that 
any unitarizable module is completely reducible. 

Proposition 4.1. The representations V^ , 2m < 2n s and V^ p , 2m , 2n j that occur in the 
decompositions of S(C d (8) C m ' n ) are unitarizable. 

Proof. We need to construct a contravariant positive-definite Hermitian form on 
C[x, T}]. We proceed as follows. 

First note that the space C[x, 77] is an irreducible representation of the direct 
sum of a Heisenberg algebra and a Clifford superalgebra with generators mapped 
to xj, r/1, tj^j, -j^jj for i = 1, ■ • • , m, k = 1, • • • , n and j = 1, • • • , d, and 1. 

Identifying this superalgebra with its image we have an anti-linear anti-involution 



HOWE DUALITY AND COMBINATORIAL CHARACTER FORMULA 



19 



given by 

This gives rise to a unique contravariant Hermitian form (-|-) on C[x, rj\ with 
(1|1) = 1. Furthermore for any non-zero monomial / G C[x, rj\ we have (/|/) > 0, 
and hence (-|-) is positive-definite. Therefore C[x, rf[, as a representation of the 
Clifford superalgebra, is unitarizable. 

Now it is easy to see, using (|4.1j) along with the formulas for Sp I, Sp A and °I, 
° A in this section, that osp(2m\2n) and spo(2m\2n) are invariant under the anti- 
involution t. This implies that the representations of osp(2m\2n) and spo(2m\2n) 
on C[x, rj] are unitarizable. □ 



5. Joint highest weight vectors 

In this section we will describe the explicit decomposition of the space S(C d <g> 
C m ' n ) under the joint actions of the relevant dual pairs. We will do so by explicitly 
finding a joint highest weight vector for each irreducible component. 

5.1. The case of (O, spo)-duality. Consider the (0(d), spo(2m\2n))-duality on 
the space S^C^ ® C m ' n ). Using the notation from Section 0] we make the identifi- 
cation of S(C d (g) C m ' n ) with the polynomial superalgebra C[x, 77] so that the Lie 
algebra so(d) and spo(2m\2n) are identified with differential operators. 

By Theorem 14.11 we only need to find the decomposition of the space of har- 
monic polynomials H into irreducible 0(d) x g/(m|n)-modules. By Theorem 13.21 
C[x, rf\ as a gl(d) x g/(m|n)-module decomposes into J2\^d ® Kn|n> w here the 
summation is over all partitions A with /(A) < d and A m+ i < n. 

Consider first the case when m > |. Take a diagram A with \[ + \' 2 < d and 
let v\ be the corresponding joint gl(d) x g/(m |n)-highest weight vector in C[x, 77] 
of the form in Theorem 13 .11 or Theorem 13 .21 Note that in this case it is automatic 
that A m+ i < n, as long asn> 1. 

Here and further we use | to denote the (m + n)-tuple (|, • • • , |; — |, • ■ ■ , — |). 
That is, the first m entries are |, while the last n entries are — |. 

Proposition 5.1. Suppose thatn > 1 andm > |. T/ie vector v\ is A-harmonic 
of 0(d) -weight corresponding to the diagram A. Therefore 

n — / , V 0(d) » ^ m |n ' 
A 

where A ranges over all diagrams with \[ + A' 2 < <i. i/ere £/ie weight A + <i| 
denotes the sum of the gl(m\n) -weight corresponding to the Young diagram A 
wzt/i i/ie (m + n) -tuple dh. 
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Proof. Note that by our choice of the Borel subalgebra of so(d), it is automatic 
that V\ is an 0(d) x g/(m|n)-highest weight vector. (In fact this is true for any 
A.) Thus in order to show that V\ is °A-harmonic it is enough to show that it is 
annihilated by °A™ = ^ - =1 ^3 J+i-j - This is because v\ is already annihilated 

by the nilpotent radical of the Borel subalgebra of gl(m\n), which together with 
°Aff generates the nilpotent radical of the Borel subalgebra of spo(2m\2n) . Also 
note that if X[ < m, then the joint highest weight vector is the usual joint 
highest weight vector in the classical 0(d) x sp(2m)-duality and hence is killed 
by °Aff [H]. So we may assume that X[ > m. In this case in order to show that 
°A^t>A = 0, we consider the classical 0(d) x sp(2m + 2n)-duality. Here the joint 
highest weight vector w\ of 0(d) x sp(2m + 2n) is a product of determinants of 
the form A^A^ ■ ■ • A^ , with only A' : exceeding m. That is, only in A^ can 

we possibly have variables of the form x 3 m+i with % = 1, • • • , n. From the duality 
in the classical case we know that w\ is harmonic and hence in particular 
°Affw\ = 0. Consider the first order differential operators Tj = Yli=i r A~^j — > 

for i — 1, • • • , n. We see that °Aff commutes with all Tj and hence 

= r x • • • r Al _ m °A° WA = ° AST! ■ ■ • T K _ m w x . 

But r x • • • T K _ m w x = (-l) A i~ OT-1 (A' 1 - m)\v x and hence °Affv x = 0. 

Finally the addition of <i| to the g/(m|n)-highest weight A is of course due to 

dnj). □ 

As in ^3] one shows that v\ indeed has 0(c?)-weight corresponding to the Young 
diagram A. But as A ranges over all partitions with X[ + A' 2 < d we conclude 
that the v\s generate the complete set of all finite-dimensional irreducible 0(d)- 
modules. Due to 0(d) x g/(m|n)-duality in H we see that 



^ = E v ow ® V r 



2 

mln ' 



where A ranges over all Young diagrams with X[ + X' 2 < d. 

Proposition 5.2. Suppose thatn > 1 andm > |. T/ien as an 0(d) xspo(2m|2n)- 
module we have the following decomposition. 



s(c d si c m '») = E y A (d) ® v; 



A+4 

spo(2m|2n) ' 



where X ranges over all Young diagrams with X[ + X' 2 < d. 

Now consider the case when m < |. For this case we introduce new even 
variables so that the total number of even variables is at least |. Since the case 
when d is odd is analogous we assume for simplicity that d is even and we add 
new variables x 3 m+1) • • • , x 3 ^, j = 1, • • • , d, to the polynomial superalgebra C[x, rj] 
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and denote the resulting superalgebra by C[x', 77]. That is, we are considering the 
embedding S(C d ® C m l n ) C S(C d ®C^ n ). Without further mentioning we adopt 
the convention of adding a ' to operators, vectors etc. when we are regarding them 
as over S(C d <S> Ca' n ). So for example we denote the corresponding Laplacian of 
C[x', 77] by °A'^, 1 < i,j ' < |, etc. and call °A / -harmonic an element / G C[x', 77] 
that is annihilated by all these Laplacians. We note that A?f = °A'^, for 
1 < i, s < m etc. Furthermore A'^ x , with either i or j not in {1, • • • , m}, is a 
sum of second order differential operators, each of them involving differentiation 

with respect to some of the new variables x 3 m+1 , ■ ■ ■ , x 3 i that we have introduced. 

2 

It follows that if / € C[x, 77] C C[x', 77], then / is °A'-harmonic if and only if / is 
°A-harmonic. Thus °H C °H'. 

Now in C[x', 77] we know that the subspace of °A'-harmonics is 

A 

where A ranges over all Young diagrams with X[ + \' 2 < d by Proposition 15.11 
Let v' x be a joint highest weight vector in °H' for the component (g) 2 . 
Then if the first column exceeds | we have up to a scalar multiple 

v' x = A 1Ai A A , ---A^. 
Otherwise we have up to a scalar multiple 

v[ = Ay Ay ■■■ Ay . 

Suppose A is such a diagram with A m+ i > n. In this case the n-th column of 
A exceeds m and hence Ay contains at least one row with entries consisting 
entirely of newly introduced variables. Now by Theorem 13 . 31 and Theorem 14. II all 
the 0(d)-highest weight vectors of highest weight A in H' are, up to a scalar 
multiple, of the form A T , where T runs over all (||n)-semi-standard tableaux. 
But then it is not hard to see that one of the rows in some Aj must consist 
entirely of newly introduced variables so that A T reduces to zero when setting 

^m+i = " " " = = 0. Since °H C °H', this implies that there are no 0(d)- 

2 

highest weight vectors of highest weight A with X m +i > ti in H. 

On the other hand if \[ + A' 2 < d and \ m +i < n, it is quite easy to see, using 
Theorem 13. 2\ that v\ (that is the 0(d) x ^Z(m|n)-joint highest weight vector in 
C[x, 77]) is annihilated by °Aff = J2j=i ~£j g^ 1 -^ ' anc ^ hence v\ is indeed °A- 
harmonic. Combining the results of this section we have proved the following. 
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Theorem 5.1. We have the following decomposition of°H as an 0(d) xgl(m\n) 
module: 



A 



where A ranges over all diagrams with X[ + A' 2 < d and A m+ i < n. Thus as an 
0(d) x spo(2m\2n) -module we have: 

S (c d ® c m '-) - Yl v o( d ) ® CAi^ 

A 

where A ranges over all Young diagrams with A^ + Aj < d and A m+ i < n. Here the 

A+d— 

labels of the spo(2m\2n) -highest weight ofV ^^n) w ^ respect to the Dynkin 
diagram of Section\2.J\ is obtained by applying \2.2}) to the gl(m\n) -weight A + d|. 



Proof. The preceding discussion already shows that the theorem hold when n > 1 
and m < |. Since in the case when m > | and n > 1 the condition A m+ i < n is 
vacuous, the theorem is true in this case due to Proposition 15.21 But of course 
the case n = is the well-known classical case for which the conclusion of the 
theorem hold as well. □ 

Remark 5.1. Partial results on the decomposition of the space S^C^ ® C m ' n ) 
with respect to the joint action of 0(d) x spo(2m\2n) were obtained earlier by 
Nishiyama in [21] by constructing certain 0(d) x spo(2m|2n)-joint highest weight 
vectors in S(C d (&C m \ n ). However, the full set of such joint highest weight vectors 
(and hence the complete decomposition) was not obtained in there. 

5.2. The case of (Sp, osp)-duality. Now consider the action of the dual pair 
(Sp(d), osp(2m\2n)) on the space S(C d <S> C m ' n ). The procedure is similar to that 
of Section 15.11 

In view of Theorem 14.21 we again only need to find the decomposition of the 
space Sp H with respect to the joint action of Sp(d) x gl(m\n). According to 
Theorem 13.21 C[x. rj\ as a gl(d) x gZ(m|n) -module decomposes into V/ <8> V^ n , 
where the summation is over all partitions A with /(A) < d and A m+ i < n. 

Let us first consider the case when m > |. We take a Young diagram A 
with /(A) < | so that the condition A m+i < n here is automatic. We recall 
from Section that the finite-dimensional irreducible representations of Sp(d) 
are parameterized by diagrams with length not exceeding |. 

Proposition 5.3. Suppose that m > | and let A be a diagram with /(A) < |. Let 
v\ G V d A (g> V\ be a gl(d) x gl(m\n) -joint highest weight vector in C[x, rf\. Then 
v\ is Sp A-harmonic of Sp(d) -weight corresponding to the diagram A. Therefore 



A+d| 



in it 
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where A ranges over all diagrams with /(A) < |. Here A + d| denotes the sum of 
the gl{m\n) -weight corresponding to A and the {m + n)-tuple d|. 

Proof. Since m>^ and l(X) < |, the g/(<i) x gZ(m|n)-joint highest weight vector 
is of the form v\ = Yli=i A a'. > that is, only the x variables are involved. Since 
the Borel subalgebra of sp(d) is contained in the standard Borel subalgebra of 
gl{d), v\ is an sp(d) x #Z(m|n)-highest weight vector. We need to show that it 
is 5p A-harmonic. For this it is again sufficient to show that v\ is annihilated 

by Sp Afl = J2]=i (^j^th - ^?h^ But this is clear by the classical 

Sp(d) x so(2m)-duality [2], because the formulas for the joint highest weight 
vector V\ and for the Laplacian Sp Aff in the classical case are identical with our 
formulas here. Now the proposition follows from Theorem 14.21 together with the 
fact that we have constructed an Sp(<i)-highest weight vector corresponding to 
every finite-dimensional irreducible Sp(d) -module. □ 

We now consider the case m < |. In this case the condition \ m +i <nis not an 
empty condition. Here we can apply the idea of Section 14.11 by inserting enough 
new even variables o^ m+1 , • • • , arjj and consider the Sp(d) x osp(<i|2n)-duality on 

2 

the space S(C d g> C^ n ). We identify S(C d <g> Ci |n ) with C[x', rf\ as before and 
regard C[x, 77] C C[x',^]. Again we will use ' to distinguish elements in C[x', 77] 
from elements in C[x, 77]. As in Section 14.11 it is easy to see that an element 
/ G C[x, rj\ C C[x', 77] is Sp A-harmonic if and only if it is p A'-harmonic and 
therefore Sp H C Sp H'. 

Now by Proposition 15.31 we have 

A 2 

where A is summed over all partitions of length /(A) < |. A joint highest weight 
vector v\ is given by YliLi and hence by Theorem the set of A T 's, where 
T runs over all (||n)-semi-standard tableaux of shape A, is a basis for the space 
of Sp (d) -highest weight vectors of highest weight A in Sp H' . 

Now suppose that A m+i > n. Let T be a (||n.)-semi-standard tableau and 
A T = Yli . It is clear that in this case one of the Aps must contain a row 
consisting entirely of newly introduced variables. But then this means that, by 
setting these newly introduced variables equal to zero, A T is zero. This implies 
that in Sp H there are no Sp(<i)-liighest weight vectors of highest weight A, and 
hence no Sp(d) -module of the form Vg p ^ can occur in the decomposition of Sp H 
with respect to the action of Sp(d). 

The above argument combined with Proposition 15.31 gives the complete de- 
scription of the Sp(d) x osp(2m|2n)-duality on the space S(C d <& C m ' n ), which we 
summarize in the following theorem. 
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Theorem 5.2. We have the following decomposition of Sp H as an Sp(d) x 
gl{m\n) -module: 

A 

where X ranges over all diagrams with l(X) < | and X m +i < n>. Thus as an 
Sp(d) x osp(2m\2n) -module we have: 

S(C d ® C-l") - Yl ® C(t|2n)> 

A 

where X ranges over all Young diagrams with Z(A) < | and A m+ i < n. .Here the 

X-^d— 

labels of the osp(2m\2n) -highest weight of V osp ( 2 m\2n) w ^ respect to the Dynkin 
diagram of Section \2.J\ is obtained by applying \2.1\) to the gl{m\n) -weight X + d\. 

Proof. Let A be a diagram with /(A) < | and A m+ i < n. In view of the discussion 
above and Proposition 15.31 it remains to prove that in the case when m < ~, the 
gl(d) x gl(m\n)-jomt highest weight vector in C[x, rj\ is indeed 5p A-harmonic. For 

this it is enough to show that it is annihilated by ^Aff = ^J=i (a^" g 3; d + 1 -j ~ 
a a " 

dxf +1 ~ j dx: 



But this is easy to see using the formula for such a joint highest 
weight vector given in Theorem 13.21 □ 

6. Character formulas for irreducible unitarizable spo(2m\2n)- 

AND osp(2m|2n)-MODULES 

In this section we give combinatorial character formulas for the spo(2m\2n)- 
and osp(2m|2n)-representations that appear in the decomposition of S , (C d ®C m ' ra ) 
of Section We shall need a result of Enright jH] which we shall recall. Before 
this we need some preparatory material. 

Consider a Hermitian symmetric pair (G, K), where G is a real classical simple 
Lie group. Let g and 6 denote the corresponding complexified Lie algebras. Fix 
a Cartan subalgebra f) of t so that f) is also a Cartan subalgebra of g. Let b be 
a Borel subalgebra of g containing f) so that q = 6 + b is a maximal parabolic 
subalgebra of g with abelian radical u. Hence as a vector space we have q = fi © u. 
Denote by A and A(6) the root systems of (g, f)) and (t, f)), respectively, and 
let A + be the set of positive roots determined by b. Furthermore set A (6)+ = 
A + fl A(t) and let p and p% denote the respective half sums of positive roots. Also 
let A(u) = {a E A\g a C u} and put p u = | Xlaeu - ^ ^ anc ^ W(^) denote the 
Weyl groups of g and t, respectively. 

Now to each A G b* one can associate a subgroup W\ of W. Since we will 
need to explicitly compute W\ later on, we will give a detailed description of it 
now. The group W\ is the subgroup of W generated by the reflections s a , where 
a G A(u) satisfying the following three conditions [ElEh 
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(i) < A + p, a >G N. 

(ii) If for some (3 G A we have (A + p\(3) = 0, then (a\(3) = 0. 

(iii) If for some long root (3 G A we have (A + p\/3) = 0, then a is a short root. 

Associated to W\ one may define a root system A A consisting of the roots 7 G A 
such that s 7 lies in W\. Now we set A A (t) = A A n A(t), A A+ = A + n A A and 
A A (6) + = A A (£) n A A+ . The group W\(t) is defined to be the subgroup of W\ 
generated by reflection along the roots lying in A A (t)+. We have a decomposition 
of the group W\ = W\(t) x W£, where 

(6.1) W{ = {w G W\ I < wp, 6c >G Z+, Va G A A (6)+}. 

Remark 6.1. Note that our definition of W A is actually the definition of W\ +p in 
[HI and |7J. 

For p, G f)* being a A(6) + -dominant integral weight we denote the finite- 
dimensional irreducible 6-module of highest weight p by V/ 1 , as usual. 

Now let A G I)* be a A(t) + -dominant integral weight. We may extend 
to a q-module in the trivial way and consider the induced representation of 
q. It is clear that contains a unique maximal submodule and hence has a 
unique irreducible quotient, which is isomorphic to the highest weight irreducible 
0-module of highest weight A. We will denote this g- module by V A . 

For ( e f)* with <(,a>el for all a G A(£), we denote the unique A(t) + - 
dominant element in the iy(6)-orbit of £ by £. 

We have the following character formula for an irreducible unitarizable repre- 
sentation Vq. 

Theorem 6.1. 017] We have 



chK 



where l(w) is the length of w in W\. 

6.1. Character formula for spo(2m|2n)-modules. It follows from Theorem l4.ll 

in the case when n = that we have the following identities of characters, for d 
even and odd, respectively. 

(6.2) 



\ 1 

(yi • -y m )i (1 _ Xiy . ){l _ g< -x y .) = E Ay o(^V^, ^ven, 

(6.3) 

d-l 

2 m j l 



=1 j=i 
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Here A is summed over all partitions with X[ + A' 2 < d such that /(A) < m and | 
stands for the m-tuple (|, • • • , ^). Let us write Xo(cZ)( x ) f° r the character of Vqu\ 
to stress its dependence on the variables X\, • ■ • , Xrdy We will now apply Theo- 
rem |0] to the Hermitian symmetric pair (Sp(2m),U(m)), so that g = sp(2m) 
and t = gl(m). We may now rewrite chV sp , 2r ^, in terms of Schur functions as 
follows. Since p u + is iy(t)-invariant, we have by Theorem 16.11 



rhV + z - (v 7, \i A+df 



^^+4^ 1 ' ) ' (U ' )s «'(^+pd)-p^ y ' ) 



where here and further p d = p + d\. Here , d i is the subset of the Weyl group 
of sp(2m) defined by (j6.1|) . 



Remark 6.2. As we now need to deal with di , H / A+d i and W x+d i($) for 

different m at the same time, we introduce a superscript m in order to distinguish 
them. So for example W'™ t is the subset W\ , t of the Weyl group of sp(2m). 

Combining this with ()6.2|) and ()6.3|) . respectively, we have for even and odd d 
respectively 



(6.4) 

d 
2 

flfld ,,„;,)( i .•,•//,) ni<^-< m (i-y^-) ' 

2 m 



i m i S^ew*." 1 ^ 1 )' ( " ,)s «,(A+p d )- Pd (y) 

d-1 

m j 



(6-5) 



E[l<i<j<m(l — ViVj) 



Here Xo(<z)( x ) = Xo( d )(x) if and only if A is obtained from A by replacing the 
first column of A by a column of length d — A' x . That is, the corresponding 
representation of the Lie algebra so(d) are isomorphic. Here and further we 
denote by A the Young diagram obtained from A by replacing its first column by 
d — \[ boxes. 

In order to distinguish such representations at the level of characters in the 
case when d is odd let us take —I G 0(d) \ SO(d) and let e denote the eigenvalue 
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of —I so that we have e 2 = 1. We may then rewrite (|6.5|) as 



d-l 



niLr 



, /= J(1 - ex iyj )(l - e Xi ^(l - e % ) 

^™ ew t^i (-i) /(u,) s W ( A +p d )-p d (y) 

(6-6) E^w( e ' x ) TT (l-vv) ' 

A i Ll<i<j<m\ L yiyj) 

where now Xo(d)( e > x ) ^ s a polynomial in x and e such that when setting e = 1, we 
obtain Xo(d)( x )- Now it is easy to see that if A is a Young diagram and Xso(d)( x ) * s 
the corresponding SO^-character of Xo (d) (e,x), then Xo(d)( e ' x ) = e ' A| Xso(d)( x )> 
where |A| is the size of A. Hence we have Xo(d)( e > x ) = e Xo(d)( e ' x )- 

The identites (16 .4j) and (|6.6|) will be our starting point for a character formula 
for unitary spo(2m|2ra)-modules. We need the following lemma. 

Lemma 6.1. Suppose that f x {y) and g x (y) are power series in the variables y. 

(i) Suppose that d is odd and 

(6.7) £/*(y)Xo«)fe*) = ^ A (y) X A (d) (e,x), 

A A 

where the summation is over the full set of irreducible finite-dimensional 
characters of 0(d). Then f x (y) = g x (y), for all A. 

(ii) Suppose that d is even and 

E/ A (y)^ w ( x ) = 5> A (y)*°w( x )' 

A A 

where the summation is over the full set of irreducible finite- dimensional 
characters of 0(d). Then f x (y) + / A (y) = g x (y) + g~ x (y). 

Proof. We shall only show (i), i.e. for d odd, as the case of d even is analogous 
(in fact easier). The argument is similar to the that of [3J. 
We multiply the identity ()6.7|) by the Weyl denominator D of the Lie group 

SO(d) and using the Weyl character formula for Xso(d)( x ) = — — ~D 

we obtain 

(6.8) ^/ A (y)e |A| ^(-l)'H e ™(A+/>) = ^ A (y) e l A l ^ (-l)'M e w{x+p) . 

A w€W A w£W 

Now as A ranges over all integral dominant weights, A + p ranges over all regular 
integral dominant weights of SO(d). Hence if A ^ \x as S'0(c?)-dominant weights, 
then the set of weights {w(X + p),w(p + p)\w G W} are all distinct. Clearly two 
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weights A and \i are equal as S , 0(c?)-dominant weights if and only if /x = A. Thus 
looking at the coefficient of e x+p in ()6.8|) we obtain 

e |A| /A(y)e A+ " + el A ' A(y)e A+ " = ^g x (y)e x+p + ^g- x (y)e x+p . 
Since e' A 'e' A ' = e, we conclude that / A (y)e A+p = g(y)e x+p and hence / A (y) = 

From the identities ()6.6|) and (|6.4|) by using Lemma fo.ll we obtain the following 
results for every m G N: 
In the case when d is odd 



w£W l,m+l 
A + d£ 



E ^ ^ ^ S u)(A+p d )-p d (^l' ' " ' Vm) 



A + d^ 



In the case when d is even 

E (- 1 )^ )s MA^)-p d (^>---^-°) 



_TirE , 771 + 

«>eVv i 



2 



iiT-t ,m+l 
A+d^ 

(A+p d )-p d (^l' ' " iVm) 



A + d^ 



W&W-' i 

A+di 



This allows us to define, in the case when d is odd, an element S x p (yi, 2/2, • " " ) in 
the inverse limit of symmetric polynomials, that is uniquely determined by the 
property that 



A + d 



Similarly we may define an element S x p (yi,y2, ■ ■ ■ ) + S x p (y\, y 2 , • ■ ■ ) in the case 
when d is even. 

Remark 6.3. The elements S x p (yi,y2, • • • ) and S x p (yi, y 2 , • • • ) + S x p (yi, y%, • ■ ■ ) are 
in general infinite sums of symmetric functions and hence are strictly speaking 
not symmetric functions. However, in these infinite sums there are only finitely 
many summands for any fixed degree. 
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We now take the limit as m — > oo in ()6.4|) and ()6.6|) and obtain the following 
identities, respectively. 



(6.9) 



d 

2 OO 



Ma 



1 



(6.10) 

1 



X>°( 



s A (y) 



Hi<i<j0- -viVj) 



d— 1 
2 CO 

J[ I! (1-^(1 - ex" 1 % )(l - e % ) 



ViVj) 



where A is summed over all 0(cZ)-highest weights and y = (2/1, 2/2? ■ * * )■ 

The identities f!6.9|) and ()6.10|) follow from the fact that setting y m+ i = y m +2 = 
• ■ • = 0, they reduce to identities (|6.4|) and (|6.6j) . respectively. Thus the left-hand 
sides and the right-hand sides of ()6.9|) and (j6.1()j) give rise to the same elements 
in the ring of the symmetric functions, respectively. 

Recall that u, the involution of the ring of symmetric functions which sends the 
complete symmetric functions to the elementary symmetric functions, is defined 
by (^dX/eN = rijeN^ + w j) ( see f° r example [24J). We can now apply u 

partially to the variables y m+ i, y m +2, • • • ■ After that we set the variables y m + n +i = 
y m+n+ 2 = • • • = and we obtain the following identities (z\ = y m +i, for I = 
,n). 



d 

2 m n 



n n n~ + x * zi ^ 1 + Xi lzi ^ 



7=1 7=i f=i t 1 ~ ^fiX 1 - x i 

HS^ p (y,z)U i>l (l + y i zi) 



(6.11) =J2*o(d)( : 



Ul<i<j< m 0- " ViVj) Y\l<Kk<n( l ~ Z l Z k) ' 



d-1 

2 m n 



tttttt i1 + eXjZi)(l + tx, L 1 z l )(l + ezj) 
fi ji t\ (1 - e*y;)(l " ex^ yj ){\ - ty 3 ) ~ 
( ,, , ST a , HS^y^)Y\ hl (l + y lZl ) 
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Remark 6.4. We note that uj(JJ 
the following identities: 



i 



) = n 



i 



This follows from 



l<Z<fc l-z ; z fe 



l<Z<fc 1 — 2;2 fc ' 




n t^=£*. 

<l<k fi 




where A is summed over all partitions with even row lengths, and /x is summed 
over all partitions with even column lengths. 

Let us now explain the term HS^ p (y, z). Since setting the variables y m + n +i = 
y m+n +2 = ■ • • = the expression S^ p (y) reduces to a finite sum whose summands 
are Schur polynomials with coefficients ±1, it follows that applying the involution 
u to it, we obtain a sum whose summands consists of hook Schur functions with 
coefficients ±1. In fact if y 2 , ■ ■ ■ ) = J2u £m s mO/i> 2/2, ■■■), where e M = ±1, 



where HS^yi, ■ ■ ■ , y m ; zi, z 2 , ■ ■ • ) is the hook Schur function of pQ in the vari- 
ables yi, ■ ■ ■ ,y m and zi, z 2 , ■ ■ ■ corresponding to the partition \i. Next setting the 
variables z n+ i = z n+2 = • • ■ = we get the hook Schur polynomial associated to 
/i, which we denote by HS^yi, ■ ■ ■ , y m ; Z\, ■ ■ ■ , z n ). One property of hook Schur 
polynomials is that HS^yi, • • - , y m \ Zi, ■ • ■ , z n ) is non-zero if and only if \i lies 
in the (m|n)-hook, i.e. /i m+1 < n. So if S^(y) = ^ e M s M (y), then by HS* p (y,z) 
we mean the expression 



Therefore A in (j6.11J) and ()6.12|) is summed over all 0(<f)-highest weights A such 
that A m+ i < n. 

From Theorem 15.11 Lemma 16. 11 and the identities (|6.11j) and (|6.12j) we obtain 
the following theorem. 

Theorem 6.2. Let X be a diagram of Theorem I5.il and let V spo( ^ m ^ 2n ^ be the 
irreducible spo(2m\2n) -module corresponding to Vo(d) un & er the Howe duality. 
Here | is the (m + n)-tuple (|, • • • , |; — |, • • • , — ~). 
(i) If d is odd, then 



then (cf. 0) 



CO 



(SspiVl, 2/2, ■■■))= ^2 e U- HS U-(yi' ■'■ iVm\ Z U Z 2 , ■ • ■ ) 
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(ii) If d is even, then 



mV sno(2m\2n) + CJ1 V svo(2 m \2n) ~ 1 " = 



spo{2ml2n) spo{2ml2n) w " ' ni<^< m (i - w) rw<„(i - ^) 



i/ere yz 1 stands for the product y± - • • y m Zi 1 • • ■ z n 1 . 

Remark 6.5. The expression HS^ p (y,z) in general involves an infinite number of 
hook Schur functions, so the computation of these characters is a highly non- 
trivial task. In order to have a method to compute them, it is necessary to have 
an explicit description of W. ,m , 1 . We will do this in Sectional From this we 

will then show in Section |H] that the coefficients of the monomials in a character 
of a fixed degree can be computed by computing a finite number of hook Schur 
functions. 

6.2. Character formula for osp(2m|2n)-modules. As the arguments in this 
case are very similar to the one given in the previous section, we will only sketch 
them here. 

It follows from Theorem 15.21 in the case when n = that we have the following 
identity of characters. 

d 

2 m i 

(«3) (». nil (1 - JJW)(1 - x: ^>c<s. 

Here A is summed over all partitions with l(X) < min(|,m) and | stands for the 
m-tuple (|, • " ' > §)■ Let us write Xs P (d)(y) ^ or ^ ne character of Vsp( d) ^° s ^ ress 
its dependence on the variables y±, ■ ■ ■ ,yd. We now apply Theorem 16.11 to the 
Hermitian symmetric pair (SO*(2m), U(m)), so that we have q = so(2m) and 

6 = gl(m). By Theorem 16.11 we can then write chV^^m) in terms of Schur 
functions as 



. , ,1 , ' <WHYV 

chV so(2m) =[X 1 ---X. 



w{\+Pd)-Pd K 



Here W*™ k is a subset of the Weyl group of so{2m). Thus we have the following 
identity. 

(6.14) 

d 
2 



10 ■<■,,,,)( i ■>;„, ■) ^ x>/ ' : ""-' II ,(i -.'V',: 



=1 i= 

Analogous to the proof of Lemma 16.11 one proves the following lemma. 
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Lemma 6.2. Suppose that / A (x) and g A (x) are power series in the variables x 
and suppose that 

(6.i5) E/ A ( x )xWy) = 5> A (y)xWy)' 

A A 

where the summation is over the full set of irreducible finite-dimensional charac- 
ters of Sp(d). Then / A (x) = g A (x) ; for all X. 

From Lemma [6.21 and the identity (jbM4j) it follows that 



\ + di 



22 ( i) l( - w) s W (x+ Pd )- Pd ( x i' " ■ ' x »o> 



\ + di 



which then allows us to define an element Sg (x\,X2, • • •) in the inverse limit of 
symmetric polynomials, uniquely determined by the property that 



Xr, 



Taking the limit as m — > oo ()6.14|) and Lemma lfi~2l imply the following identity. 

d 

2 OO -, q\ I \ 

( 6 - 16 ) nil 71 Vy ^ = E*Wy) n n r • 



i=i j 



We apply to ()6.16|) the involution of symmetric functions to partially to the vari- 
ables x m+ i, x m+2 , • ■ ■ , then set the variables z n+ i = z n+2 = • • ■ = 0. We arrive at 
the following identity (zi = x m+ i, for I — 1, 2, • ■ ■ ). 

d 

( } = L^n (i-xx)u d-z^y 

x LLl<i<j<rri\ L ^^j) lll<l<k<n\ L ^k) 

where HS^ Q (xi, • ■ ■ , x m ; z\, • • • , z n ) is obtained by applying the involution lo to 
Sg Q and setting the variables z n+ i = z n+ 2 = • ■ ■ = 0. As before it is also a sum 
whose summands consist of hook Schur polynomials with coefficients ±1. By 
Theorem 15.21 and Lemma 16.21 we then obtain the following theorem. 
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Theorem 6.3. Let X be a diagram of Theorem \5.2l and let V osp ( 2 m\2n) ^ e ^ e 
irreducible osp{2m\2n) -module corresponding to Vsp(d) un der the Howe duality. 
Here \ is the (m + n) -tuple (|, • • • , |; — |, • • ■ , — ~). Then 



where xz" 1 denotes the product 

Remark 6.6. We actually have Howe dualities of the dual pairs (0(d), 0(C OO )) 
and (Sp(d), fl(Doo)) on the space 5(C d (g> C°°). Here the infinite-dimensional Lie 
algebras g(C 0O ) and g(-Doo) are Kac-Moody algebras corresponding to the in- 
finite affine matrices and D^, respectively [TH]- From these dualities one 
can show that, using similar arguments as we have given here, the correspond- 
ing characters of those irreducible representations of 0(C DO )- and 0(-Doo )-modules 
are given by certain infinite sums of symmetric functions. Applying the involu- 
tion uj to these characters one obtains the characters for our spo(2m\sn)- and 
osp(2m|2n)-modules. Thus the characters of the representations of 0(6*00) (re- 
spectively 0(-Doo)) that appear in these dualities determine the characters of the 
representations spo{2m\2n) (respectively osp{2m\2n)). 



7. The Group W x+d i 

Throughout this section A = (Ax,A 2 , ••• ,X S ) is a partition of non-negative 
integers of length s < d. We shall describe the groups W™ rfX and W'^ d i (t) for 

the Hermitian symmetric pairs (Sp(2m),U(m)) and (SO*(2m),U(m)). 

Recall that the group W™ d i is defined as the subgroup of the Weyl group of 

sp(2m) or so(2m) generated by reflections corresponding to a G A(u) satisfying 
conditions (i), (ii) and (iii) given in Section We will simply refer to them as 
conditions (i), (ii) and (iii) in what follows. 

7.1. The case of 0(d) x sp(2m)-duality for d even. In the case when d is 
even W™ ,1 is the subgroup of the Weyl group of sp(2m), which is isomorphic 

to the sign permutation group S m x Z™. The positive roots A + of sp(2m) are 
generated by the simple roots — 2ei,ei — e 2 ,e 2 — e 3 , • • • , e m __i — e m . We have 
p = —ei — 2e 2 — • • • — me m , which we write as 

P= (-1,-2, ••• ,-m). 

We have the condition that X[ + A 2 < d. Now A + (6) is generated by the simple 
roots e% — e 2 , • • • , e m -i — e m , while A(u) consists of roots of the form — e, — €j, 
1 < i < j < m. 
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Let us first consider the case s — |. In this case 

A + d^ + p = (Ai + - - 1, A 2 + - - 2, • • • , A|_ x + 1, A|, -1, -2, • • - , -m + -). 

We see that A + d| +p has no zero coefficient, and hence condition (iii) is vacuous. 
It follows that for each i = 1, • • • , ~ with m > Aj + d — i we have 

(7. 1) (A + cfi + P> -e< - eA !+ d-i) = 0. 

On the other hand if m < Aj + d — i, i — 1 • • • , ~, we have for all i = 1, • • • , m 

(7.2) (A + rfi + p, -e, - e 4 ) < 0. 

This implies by condition (ii) that if a = — — t\ is such that s a G VKj^i, 

then neither k nor / can be in the index set J = {1, • • • , |, Ai + d — 1, A2 + d — 
2, ■ • • , Ad + |}. Let 1° = {1, • • • , m} \ J. Let a = -e k - e t with k, I G J°. Clearly 
we have 

< A + d- + p, a >G N, 

and hence condition (i) is satisfied for such an a. This implies that W™ ,1 is 
generated by the reflections s a with a = — — €1, k,l & 1° . Hence ,1 is the 
sign permutation group on the index set 1°. Therefore , x (6) is equal to the 
permutation group of the index set 1° and hence A x+d i(£) + consists of — t\ 
with < / and k, I G 1°. 

Next consider the case s < |. In this case we have 

1 d , d d d N 

A + d- + p= (Ai + --1,--- ,A s + --s,--s-l,-- - ,^0^,-1,-2, ••• ,-m+~). 

d 
_> 

Since (A + d| + p, 2ed) = 0, condition (iii) implies that if a G A(u) is such that 
s a G W" 1 1, then a is a short root. As in the previous case (|7.1|) and ()7.2|) hold 



' 2 



in this well with i = 1, • • • , s. In addition we have for j = s + 1, 

(7.3) (A + d~ + p, -6j - e d _j) = 0. 

Let J = {1, • • • , d — s — 1, Ai + d — 1, • • • , X s + d — s} and J - = {1, • • • , m} \ J. 
Similarly as in the previous case conditions (i) and (ii) now tell us that a G A(u) 
is such that s a G W^ d i if and only if a = — — q with k,l G I~ and k 7^ /. 

Clearly W™ +d i is equal to the even sign permutation group (i.e. permutations 

with an even number of sign changes) in the index set I~ . Therefore W^_ d i(t) 
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is the permutation group on the index set I~ and hence A x+d i(t) + consists of 
e& — ei with k < I and k,l G I~ . 

Finally consider the case when s > |. In this case we have 

1 d 1 x d d 

A + d- + p =(Ai + - - l, • ■ ■ , \ d - s - - + s, s - -, 

d N 

1 , 0, ■ • • , 1 + - — s, — 1 + - — s, • ■ ■ , — m + -). 




2' 



2 s s+1 

Then (|7.1j) and (|7.2j) hold for z = 1, • • • , d — s and we have in addition 
(7.4) (A + d^ + p, -6j - e d _ i+2 ) = 0, j = d-s + 2,-.. , -, 



(A + rf- + p, -2ed +1 ) = 0. 



,1 
2 

Let 1+ = {d - s + 1, s + 1, s + 2, • • • , m} \ {Ai + d - 1, • • • , A d _ s + d - (d - s)}. 
Then W™, i is generated by s a , where a = — e k — e t with k,l & I + and k ^ I. 

A+d 2 

This implies that W^_ dl is the even sign permutation group on the index set 
J + and hence W™,i (t) is the permutation group on the index set I + and hence 

A+a 2 

A A+d i (t) + consists of — q with < I and fc, Z G J + . 

7.2. The case of 0(d) x sp(2m)-duality for d odd. Suppose that s = ^-k We 
have 

. .1 d . 3 3 13 d. 

A + d- + p = (Ai + - - 1, • • • , A^3 - -, -, - -m + -). 

d+l 
2 

Therefore (jTTJ) and (T7~2l hold for i = 1, - • • , ^ and also 

(A + d — h p, — e^i — ed+a) = 0. 
2 2 2 

Note that the coefficients of A + d\ + p are all half integers. Hence if /3 G A(u) 
is a long root, then 

1 - 1 

< X + d- + p,$>e - + Z. 

^ _ 

Thus the lone roots are eliminated from the consideration of W™ ^ bv condition 

(i). Now let J = {1, • • • , 4=1, 4±2, Ai + d - 1, • • • , A d _ s + d - (d - s)} and let 
7° = {1, • • • , m} \ J. Then W™ x is the even sign permutation group on the 

index set 1° so that W™,! (t) is the permutation group on 7°. 
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Now suppose that s < ^±1 so that 
A + d- + p = 

. d d d 113 d 

d+1 

2 

Thus ()7.1|) and f)7.2j) hold for % — 1, • • • , s. Furthermore ()7.3|) holds for j = 
s+1, ■ ■ ■ , As before the long roots in A(u) are eliminated from considerations 
of W r+d±- Set J = {1, • • • , d — s - 1, Ai + d — 1, • • • , A s + d — s} and let I = 
{1, • • • , m} \ J. Then W™ d i is the even sign permutation group on the index set 
I so that W™ +dl {£) is the permutation group on I~. 
Finally consider the case when s > so that we have 

1 d , d d 

X + d- + p =(Ai + - - 1, • • • , X d - S - - + s, s - -, 

V v ' 

d—s 

1 1 d d d 
- , — , •••,H s, — 1 H s, • • • , — m H — ). 

\_JJ 2 2 v ^2 2 ; 

d+1 d+3 s S+1 

2 2 

Thus dHJ and (Q still hold with z = 1, • • • , (d - s) and also (Q holds with 
j = 1, • • • , Again the long roots are eliminated from the consideration of 
W™ d i . Let J = {1, ■ ■ • , (d-s), (d-s)+2, • - • , Ai+d-1, • • • , X d _ s +d-(d-s)} 
and let J + = {1, ■ • • , m} \ J. Then is the even sign permutation group on 

the index set I + so that W^ d x(i) is the permutation group on I + . 

7.3. The case of Sp(d) x so(2m)-duality. In this case W™ d i is a subgroup 

of the Weyl group of so(2m), which is isomorphic to the even sign permutation 
group S m x Z™ -1 . The positive roots A + is generated by the simple root — e\ — 

£2, ei - e 2 , e 2 - e 3 , ■ • ■ , e m _i - e m , and hence p = -e 2 - 2e 3 (m - l)e m , which 

we write as 

p = (0,-1,-2,-.. + 
Let A = (Ai, A 2 , • • • , Ad) be a partition so that 

A + di + p = (Ai + ^, A 2 + ^ - 1, - ■ • , A| + 1,^, -1, -2, ■ • • , -m + ^ + 1). 

f+i 

The set A(u) consists of roots of the form — — ei, with k ^ I. 
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We have in the case m> \ + d — i + 2 

(7.5) (A + rfi + p, -€i - e Xi+d ^ i+2 ) = 0, z = 1, ■ • • , ^. 

On the other hand if m < A, + d — i + 2, then for every £ = 1, • ■ ■ , m we have 

(7.6) (A + d- + p, -e* - e t ) < 0. 

This implies by condition (ii) that if a = — — q with s a G ,i, then 



2 



fc, Z cannot be one of the indices in (|7.5J) and (|7.6jl . On the other hand set 
J = {i ; . . . ; |, Ai + d + 1, A 2 + d, ■ ■ ■ , Ad + | + 2} and let I = {1, 2, • • • , m} \ J. 



Clearly if a = — — q with k,l £ I, then 

(A + d- + p, -e fe - ej) G N, 

and so condition (i) is satisfied. Of course here (iii) is irrelevant, as A is simply- 
laced. Thus W™ ,i is equal to the even sign permutation group on the index set 

A + d 2 

I and hence W^_ d i(t) is the permutation group on /. 

From our explicit description of W^_ dL we have the following. 

Proposition 7.1. Let A = (Ai, • • • , A s ) and let W™ d ± be the corresponding group 

of either sign or even sign permutation group on the index set I C {1, • ■ • , m}. 
Write fi = X + d^+p = (px, p 2 , • ■ • , Pm) for the generalized partition with either 
all integral or half-integral row lengths. 

(i) We have \pi\ ^ \Hj\, for i,j G / with i ^ j . 

(ii) For w G W£, d t the rows of the generalized composition w(X + d\ + p) are 

all of different length. 

(iii) In the case s < | we have p, < 0, for i E I . 

(iv) In the case s > | we have p^ > and i G / if and only if i — (d — s + 1) 
and p d - s+1 = s - f . 

(v) For allmeN we have WV\.x C . 

8. Consequences for the character formula 

In this section we will use the result of Section[7|to study the character formulas 
of Section El In Section [7| we gave a description of W*,!. However, in the 

A +"2 

character formula we actually need to have a description of W^ d x ■ 

Let w G W*, d i . From Section [7| we know that w is either a sign permutation 

or an even sign permutation on an index set I. 

Recall the decomposition W* +i i = W* +dk (t) x W^ dk . As W^ +dL is the sign 

or the even sign permutation group on I and W* ,i (6) is the permutation group 

A ' "7 
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on /, it follows that the elements of W' jX are in one-to-one correspondence with 

A-hcfjT 

either the sign or the even sign changes of the index set /. This correspondence 
can be made explicit as follows. Let I = {ji < J2 < ' • • < jt} an d set p = 
(fix. Pi-, " " " > Pk) and pi = (p^, pj 2 , ■ ■ ■ ,Pj t ). Let r be either a sign change or an 
even sign change of /. Let a be the unique permutation on / which permutes the 
rows of the generalized composition r(p/) so that ar(pi) is a generalized partition. 
Set w T = <jt, then w T G W' , ± is the element corresponding to r under the above- 

mentioned one-to-one correspondence. More explicitly, if r changes the signs of 
pi at the rows i\ < %i < • ■ • < ii, then a is the permutation that moves ii to 
ji, to jii • • • , «i to ji. After that the remaining indices • • ■ ,j t are then 
assigned from the indices I \ ■ ■ ■ , ii} in increasing order. 

We are now in a position to describe w(X + d\ + p) with w G W^* di _. For 

this it is convenient to identify W.'i with either a sign change or an even sign 

A +"2 

change of the index set I — {ji < 32 < ■ ■ • < jt}- Set 

\ + d^ + p = p= {pi,p 2 , ■ ■ ■ ,Pk)- 

Let w G W^* d i + and let t w be the corresponding sign change. Let us suppose 
that t w corresponds to sign changes of the subset I w C J. Suppose that I w = 
ii,i 2 , ■ ■■ Then 

T w (\ + d^- + p) = (p J i,p 2 , ••• >-f*h,'-- 

That is, r w (p) is obtained from p by replacing all the rows indexed by I w with its 
negative. Set a w equal to the unique permutation on {1, 2, ■ • • ,k} that permutes 
the rows of generalized composition r w (p) so that the resulting is a generalized 
partition. We denote by A W (X + <i| + p) the partition cr w r w (X + p) — p — d-^. 

The following proposition is an easy consequence of the correspondence between 
sign changes of the index set I and W , x . 

A +"2 

Proposition 8.1. With the notation introduced above we have 



A«,(A + t2~ + p) = to(A + + p) - p - d^. 
Using Proposition 18.11 we can now prove the following corollary for the char- 
acters of V s ^(2m\2n) an< ^ Ksp(2m|2n)- R- ecan the character formulas given in Theo- 
rem 16.21 and Theorem 16. HI In these formulas the expression H S w ^ x+d i +p ^ _ p _ d i is 

the hook Schur function associated to the partition w(\ + d\ + p) — p — d\. Due 
to Proposition 18. ll we will from now on write HS Aw ( X+d i +p ^ for H S w ^ x+d i +p ^ _ p _ d i ■ 

The next corollary shows that in general the character formulas involve an 
infinite sum of hook Schur functions. 
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Corollary 8.1. Fix a diagram X corresponding to V^*^ or V^ M2n) of 
length l(X) = s. 

(i) Suppose that m > s + 2 and n > 1. Then HS^ Q and HS^ p are infinite 
sums of non-zero hook Schur polynomials. 

(ii) Suppose that s = m and X s > n. Then HS^ and HS^ p are finite sums of 
non-zero hook Schur polynomials. 

Proof. Take any k > Ai + d so that both k and k — 1 lie in the index set / 
associated to W* +d i of sp(2k) or so(2k). Let w G W^*i correspond to r w , the 

even sign permutation that permutes the indices k — 1 and k and changes the 
signs of them. It is then easy to see that A W (X + d\ + p) is a partition with 2 
from the (s + 3)-rd row on. Since n > 1, it follows that the partition associated 
to A w (\ + + p) lies in the (mjn)-hook and thus its corresponding hook Schur 
polynomial is non-zero. This proves (i). 

To prove (ii) let k > Ai + d and consider any w G W^ +dl corresponding to 

to a sign change involving k. Then A UI (A + d-^ + p) is a partition with the first 
m + 1 rows exceeding n. But then the corresponding hook Schur polynomial is 
zero. □ 

Let C[[y, z]] denote the ring of power series in the variables y and z. We have 
a natural filtration of ideals determined by the leading term. 

C[[y,z]] 3^2 ••• . 

The formulas of Theorem 16.21 and Theorem 16 .HI involve in general an infinite 
number of hook Schur polynomials. However, for a fixed monomial that appears 
in the character formula we can use a finite number of hook Schur polynomials 
to compute its coefficient. This follows from the following proposition. 

Proposition 8.2. Let X be a partition with /(A) = s and X[ + X' 2 < d. Let k > d. 

(i) // s > \ we let I = 2k + |A| - 2s - 1. // s < f we let I = 2k + \X\ - d. 
Then we have 

F^(y,z)= (-^^(A^I+dCy;*) (niodff,)- 

(ii) Let I = 2k + |A| - d - 1. Then 

Proof. The theorem follows rather easily from Proposition l8.ll We will only prove 
(ii), as (i) is quite similar. We may assume without loss of generality that k G I. 
Consider w G W' ,! such that w & W[' k ~l . This means that t w changes the sign 
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of k. We consider the partition A W (X + d\ + p). It is not hard to see that the 
size of this diagram is at least 2k — d + J2i=i A* — 1. But this means that the 
hook Schur polynomial determined by A W (X + d\ + p) contains only monomials 
of degree I = 2k — d + J2i=i \~ 1- Thus the hook Schur polynomials associated 
to A W (X + d\ + p) with w G Wl'*~i contain all monomials of HS* of degree less 
than or equal to I — 1. □ 

We now compute the functions HS^ Q and HS^ p explicitly in the case of A = 
(0, 0, • ■ • ,0), the trivial partition. 

Let us first consider the case of HSg p with A being the trivial partition. We 
will write in this case simply HS sp for HSg p . In this case W* +d i is the group 

of the even sign permutations in the indices d, ■ ■ ■ ,k. Let w G Wl' k x and let 

t w be the corresponding sign changes. Let us suppose that t w changes signs at 
the following I rows: %\ < i<i < ■ ■ ■ < < i[. Here i\ > d and I is an even 
non- negative integer. Then it is not hard to see that A w (d~ + p) is the following 
partition 
(8.1) 

(i l -d+l,i l _ l -d + 2, - ■ ■ ,i x -d+l,l, ■ ■ ■ ,1, 1 - 1, • • • , 1, • • • , 1 ,0, ■ ■ ■). 

That is, the first I entries are i\ — d + 1, ii-i — d + 2, ■ ■ ■ , i\ — d + I, followed by 
ii — 1 entries of I etc. The length of A w (d^ + p) is %i. For a sequence of positive 
integers 1 — {%\ < %i < • • • < %{\ with i\> d denote by pi the partition (|8.1jl . 
Furthermore we let | J| denote Z + X^-=i b'- 

Proposition 8.3. VKe have 

HS sp {y x ,--- ,y m ;z x ,--- ,z n ) = y](-l) |f| ffg w (yi, • - • ,y m ;zi,--- ,2 n ), 

where the summation is over all tuples I = (ii,i2, • • • , ij) wi/i / even and d < 
ii < «2 < • • • < k satisfying one of the following conditions. 

(i) In the case when n > m we have I < n and at most m of the ij 's exceed 
d + n — m—1. 

(ii) In the case when m > n we have I < m. If in addition we have l+i t —t+l < 
m + 1 < I + i t+ i —t — 1, for some t — 0, 1, • • • ,1 — 1, then I — t < n. (Here 
by definition iq—Q.) 

Proof. First we note that if w G W ' , t and r m its corresponding sign changes at 

A + d 2 

the rows i x < i 2 < ■ ■ • < ii-i < ii, then = (-1) |J| . 

(i) Obviously if / > n, then the corresponding partition pj cannot lie inside the 
(m, n)-hook. Thus the corresponding hook Schur polynomial is zero. Also clearly 
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if I < m, then pi lies in the (m, n)-hook. Now suppose that m < I < n and we 
have 

ii < i 2 < ■ ■ ■ < ii-m < k-m+i <■■ ■ <ii- 
Then pi lies in the (m, n)-hook if and only if i\- m — d + (m + 1) < n, which 
happens if and only if ii- m < d — m + n — 1. 

(ii) Clearly, if I > m, then pi does not lie in the (m, n)-hook. Now if I < m 
and %i < m, then it is easy to see that pi lies in the (m, n)-hook. On the other 
hand if m < ii, we let t = 0, 1, • • • , I — 1 be such that 

l + i t -t<m+l<l + i t+1 —t — 1. 

It follows from ()8.1|) that the (m + l)-st row of fii is / — t. Thus /i/ lies in the 
(m, n)-hook if and only if I — t < n. □ 

Consider now the case of HSg Q , where A is the trivial partition. We will again 
write in this case simply HS so for HS^ Q . Here W* +d i is the group of the even 

sign permutations in the indices d/2 + 1, d + 2, d + 3, • • • , k. Let w G W*]f\i and 
let t w be the corresponding sign changes, which changes signs at the following I 
rows: %\ < ii < ■ ■ ■ < Zj_i < ii. 

First suppose that %\ ^ d/2 + 1. In this case A w (d^ + p) is the partition 
(8.2) 

(ii-d-l,ii-i-d, ■ ■ ■ , ii - d+l - 2, 1, ■ ■ ■ , I, , I - 1 , ■ ■ • , 1, • • • , 1 ,0, • ■ ■). 

h— l i2—ii-l ij— 1 

Now if Z! = | + 1, then A w (<i| + p) is 

(8.3) (ii — d— 1, - d,- ■ ■ ,i 2 — d + I — 3, Z — 1, • • • , / - 1, 

ia-1 

,/-2 ,--- ,1,0,---). 

«3— 12— 1 i;— ii— l— 1 

Note that ()8.3j) is just ()8.2|) corresponding to the sequence i 2 < h < ■ ■ ■ < ii, 
with d + 2 < i 2 . 

For a sequence of positive integers J = {i\ < i 2 < ■ ■ ■ < i{\ with i x > d + 2 we 
let i/j be the partition ()8.2|) . Let | J| = I + Yjj=i h- 

Proposition 8.4. We have 

HS so (y u • ■ ■ , y m ; z u ■ ■ ■ , z n ) = s ^(-lf A HS Uj (y l , ■ ■ ■ , y m ; z u ■ ■ • , z n ), 

j 

where the summation is over all tuples J = (ii, i 2 , ■ ■ ■ , i{) with d + 2 < i\ < i 2 < 
■ ■ ■ < ii satisfying the following conditions. 

(i) In the case when n > m we have I < n and at most m of the ij 's exceed 
d + n — m + 1. 
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(ii) In the case when m > n we have I < m. If in addition we have l+i t —t+l < 
m + 1 < I + i t+ i — t — 1, for some t — 1, • • • ,1 — 1, then I — t < n. (Here 
again Iq=0.) 

Proof. As the proof is analogous to that of Proposition 18. 3| we omit it. □ 

The module V s x po(2m{2n) (respectively V x sp(2ml2n) ) with A being the trivial par- 
tition is the 0(<i)-invariants (respectively Sp(<i)-invariants) inside S(C d ® C m ' n ). 
Thus our computations of HS sp and HS so give character formulas of these in- 
variants. On the other hand we can describe the invariants, denoted by S(C d <S> 
C m\njO(d) and s(C d ®C m \ n ) Spid \ in the following different way. Since gl{m\n) com- 
mutes with 0(d) and Sp(d), S(C d ®C m l n )°W and S(C d ® C m ^ n ) Sp ^ are modules 
over gl(m\n). We have the following analogue of classical invariant theory. 

Proposition 8.5. We have the following isomorphisms of gl{m\n) -modules 

(i) S(C d ® C m ' n )°^ = Y2\ Vm\n> w h ere th e summation is over all partitions 
A with even row lengths, /(A) < d and A m+ i < n. 

(ii) S(C d ® C m l n ) 5p ^ = Kn|n' w here the summation is over all partitions 
\i with even column lengths, Z(/z) < d and fi m +i < n - 

Proof. The proof is in the same spirit as the one in the classical case given in ^3] . 
The (gl(d), #Z(m|n))-duality gives and hence taking the 0(<i)-invariants on 
both sides of (jH.lj) gives 

S(C d ® C m|n )° (d) = J2( V d )° (d) ® ^m|n- 

A 

But it is known that V d A has only 0(<i)-invariants if and only if A is an even 
partition, i.e. all rows have even length. Furthermore in this case the dimension 
of 0(<i)-invariants in V£ equals 1. This proves (i). 

For (ii) we note that V£ has Sp(<i)-invariants if and only if A has even columns, 
in which case the dimension of the invariants is again 1. □ 

As the character of the gZ(m |n)-module V^ n is given by the hook Schur function 
associated to A we obtain the following corollary 

Corollary 8.2. As gl(m\n) -modules we have 

C hS(C d <g> C m l")°W = HS ^ ■ • • > 3/m5 *1, • • • , Zn), 

A 

chS(C d ® C m \ n ) s M = HS^xx, ■ ■ ■ , x m - z u ■ ■ ■ , z n ), 

it 

where the summations over A and n are as in Proposition \8.5\ 
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From these two descriptions of the 0(<i)-invariants inside S(C d <8> C m ' n ), in the 
case when d is odd, we have the combinatorial identity 

HS x (yi, ■ ■ ■ , y m ; zi, ■ ■ ■ , z n ) = 

A 

El -\\\I\lIQ ( \( Yll<i<m,Kl<n(^ + Vi Z l) \ 
-1 1 ^HS^iV!,--- ,?/ m ;Zi,--- ,Z n )(= — -= r- r , 

where A is summed over all partitions with even row lengths, l(\) < d and 
A m +i < n, and / is summed over all I as in Proposition 18.31 with fij as in (j8.1|) . 
Similarly from the descriptions of the Sp(<i)-invariants we have 

^ ^ H S p(x\i , x m , 2^, , z n ) 

lIl<i<m,l<Kn(l + ^i^U 



y^(-l) |J| ij"5 l yj (o:i, ■ • • ,x m ;zx, - ■ ■ ,z n )L 



Ill<i< 3 <m,l<KKfi(^ XiXj)(l ZiZk) 



where fi is summed over all partitions with even column lengths, /(/i) < d and 
Hm+i < n, and J is summed over all J as in Proposition 18.41 with vi as in (|8.2J) . 



9. Tensor product decomposition 

As another application of Theorem 15.11 and Theorem 15.21 we derive in this 
section formulas for the decomposition of tensor products of two representations 
of either spo(2m\2n) or osp(2m\2n) that appear in the decomposition of S(C d <g> 

We first recall two Howe dualities involving the dual pairs (0(d), so(2k)) and 
(Sp(d), sp(2k)) on the space k(C d ®C k ). 

Theorem 9.1. [H] The pairs (0(d), so(2k)) and (Sp(d) , sp(2k)) form dual pairs 
on the space A(C d (g)C fc ). Furthermore with respect to their joint actions we have 
the following decompositions: 

(9-1) A(C d ® c k ) - Yl v m ® , 

A 

(9-2) A(C d ®C k )^J2 V sm ® <(2 fc f > 

where in the first sum A is summed over all diagrams with l(\) < d, \' x + \' 2 < d 
and Ai < k, while in the second sum \i is summed over all diagrams with l(fi) < 
d/2 and Hi < k. 

Remark 9.1. We regard so(2k) = osp(2k\0) and sp(2k) = spo(2k\0) and hence 
the labellings of their highest weights are as in Section |2~H 
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Consider for positive integers d and r the decompositions S(C d <8> C m ' r 



E M Vfa ® C(!|2n) ^d S(C ® = E 7 Vg (r) (S> CAl^r We have 

S(C d ® C m '") (8 S(C ® - £ ^ (9 v£$U ® £ V 7 r) (8 C(l|2n) 

A* 7 
= (^0(d) ® ^O(r)) ® (Xpo(2m|2n) ® Kpo(2m]2n))- 



M,7 



Now writing V^ 2n) ® v£&m = Ea^CSSS we have therefore 

(9.3) s(c d ® c-i") ® s(c ® c m '") - £ C f (v* d) ® vg (r) ) ® CSi) ■ 

On the other hand we have 

5(C d ® C m|n ) ® 5(C r ® C m|n ) S(C d+r ® C m|n ) 

~~ K 0(d+r) ^ % ( 2m |2n)- 

A 

If we let V£ {d+r) = E^Ki V o(d) ® *o( P )> that is > we re g ard ^(d+r) as an 
0(d) x 0(r )-module in the obvious way, then we have 



(9.4) S(C d ® C m l n ) ® S(C7 ® C m l n ) 6 w ( y o(d) ® ^J(r)) ® K< 



A+(d+r)| 
spo(2m|2n) ' 



Combining and we see that c^ 7 — 



M7' 



This connection between the branching coefficients and the tensor product 
coefficients, which may be regarded as a special case of Kudla's seesaw pairs j20] > 
is of course known |14j . 

Now the same argument applied to the first dual pair of Theorem 19.11 tells us 



that = a^ 7 , where 



V so{2k) ® V so(2k) — / „ Q A 



l A ^so(2A;) 
A 

Taking account the fact that the 0(d)-, 0(r)- and 0(2d)-modules that appear 
in the various decompositions may not be identical we have proved the following 
theorem. 

Theorem 9.2. Let // and 7 be diagrams lying in the (m\n)-hook and satisfying 
the conditions & + /x' 2 < d and 7i + 7 2 < r. Let V^ m ^ n) ® KpJ 2 m|2n) - 

Ea cfCS?! ■ ^ * > max (^> 7i) «nd V^f ^ = Ea <V^ +r) \ 
Then for A Zyin<7 in the (m\n)-hook with X[ + X' 2 < d + r we have c^ 7 = a^ 7 . 
Otherwise c^ 7 = 0. 
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We can derive the following theorem for osp(2m|2n)-modules in a completely 
analogous fashion. 

Theorem 9.3. For d and r even let /i and 7 be diagrams lying in the (m\n)-hook 
wtih < d/2 and < r/2. Let V^ 2n) ® V^ 2n) *Z Ea^CS^ 

Let k > max(/i 1 , 7l ) and V^f ® V^f £ A a^V^Jf* *. Then for X lying 
in the (m\n)-hook with l(X) < (d + r)/2 we have c^ 7 = a^ 7 . Otherwise c A 7 = 0. 

Remark 9.2. Of course the computation of the coefficients a^ 7 are in general 
rather difficult. There are combinatorial algorithms that in principle can be used 
to compute them. See for example ["["""] and "231 and references therein. It turns 
out that the coefficients can be computed once the usual Littlewood-Richardson 
coefficients (for the general linear group) are known. The precise formulas are 
given in "2"Tj . 

Remark 9.3. The tensor product decompositions of the spo(2m|2ra)-modules and 
the osp(2m|277,)-modules that appear in the decomposition of S(C d ® C m ' ra ) are 
stable in the following sense. The coefficients c^ 7 are independent of m and n for 
n > 1 and m > d/2. This follows from a minor modification of our argument 
above. 

Remark 9.4. The above method for computing the tensor product decomposition 
using Howe duality appears to be quite general and could have further applica- 
tions. For example, using the gl(d) x gl(m\n)-Howe duality of Section El one can 
derive rather easily the fact that the multiplication rule of the Hook Schur func- 
tions is the same as that of ordinary Schur functions. This was derived earlier in 
[2*9*] using purely combinatorial methods. 
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